02.12.2021 17:25:00,024 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:00,050 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 83 02.12.2021 17:25:00,054 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:00,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:00,132 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:00,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:00,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,133 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 226 02.12.2021 17:25:00,159 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:00,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:00,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,173 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 185 02.12.2021 17:25:00,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:00,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,196 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:00,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:00,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:00,196 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,222 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,222 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 125 02.12.2021 17:25:00,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:00,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,241 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:00,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,257 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,257 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:00,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:00,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,274 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:00,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:00,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:00,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 85 02.12.2021 17:25:00,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:00,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 25 02.12.2021 17:25:00,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,352 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,360 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:00,360 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:00,360 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 185 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,360 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=687) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=185,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null} 02.12.2021 17:25:00,360 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,360 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 185 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,361 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=663) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=185,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null} 02.12.2021 17:25:00,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 314 02.12.2021 17:25:00,362 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=663): org.apache.kafka.common.requests.HeartbeatResponse@53675ccc 02.12.2021 17:25:00,362 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:00,362 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 185 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:00,362 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:00,362 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:00,362 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,362 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=664) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:00,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 312 02.12.2021 17:25:00,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 272 02.12.2021 17:25:00,363 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=687): org.apache.kafka.common.requests.HeartbeatResponse@204cbdec 02.12.2021 17:25:00,363 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:00,363 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 185 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:00,363 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:00,364 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:00,364 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,364 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=688) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:00,364 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 269 02.12.2021 17:25:00,423 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:00,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:00,460 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:00,461 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:00,524 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:00,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:00,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:00,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,634 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,634 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:00,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:00,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,677 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,677 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,697 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:00,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:00,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:00,698 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,724 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,724 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,724 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,724 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,724 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,725 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,725 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,742 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59753ms for 60000ms of delay, sleeping for 257ms 02.12.2021 17:25:00,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,744 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,744 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,744 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,745 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:00,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,759 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,759 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,759 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,759 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,759 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,759 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:00,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,777 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:00,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:00,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:00,778 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:00,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:00,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,852 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,853 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,926 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:00,953 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:01,006 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59040ms for 60000ms of delay, sleeping for 970ms 02.12.2021 17:25:01,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:01,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:01,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:01,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,135 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:01,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,178 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,178 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:01,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,199 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:01,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:01,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:01,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:01,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,225 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,226 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,226 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,226 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,226 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,226 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,247 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,247 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:01,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:01,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,259 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,279 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:01,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:01,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:01,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:01,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:01,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,353 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:01,454 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:01,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:01,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:01,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:01,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:01,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,637 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,643 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:01,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,679 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,679 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,700 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:01,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:01,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:01,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:01,731 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,734 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,734 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,734 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,734 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,734 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,734 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,734 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,749 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:01,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,762 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,758 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:01,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:01,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:01,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:01,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:01,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:01,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,867 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 414 02.12.2021 17:25:01,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:01,973 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 308 02.12.2021 17:25:01,981 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59306ms for 60000ms of delay, sleeping for 704ms 02.12.2021 17:25:02,035 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:02,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 208 02.12.2021 17:25:02,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:02,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:02,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,153 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,177 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 104 02.12.2021 17:25:02,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,180 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:02,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,210 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:02,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:02,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:02,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:02,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,235 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,236 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 23 02.12.2021 17:25:02,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:02,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,253 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,263 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,263 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:02,263 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:02,263 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=649) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:02,263 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 473 02.12.2021 17:25:02,263 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=649): org.apache.kafka.common.requests.HeartbeatResponse@64d0e67a 02.12.2021 17:25:02,263 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:02,263 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 473 02.12.2021 17:25:02,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 472 02.12.2021 17:25:02,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:02,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,269 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 4 02.12.2021 17:25:02,281 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,282 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:02,282 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:02,282 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=649) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:02,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:02,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,282 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:02,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:02,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:02,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:02,283 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=649): org.apache.kafka.common.requests.HeartbeatResponse@5e190970 02.12.2021 17:25:02,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,283 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:02,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:02,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:02,364 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 372 02.12.2021 17:25:02,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,367 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,367 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 272 02.12.2021 17:25:02,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:02,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 171 02.12.2021 17:25:02,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:02,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,654 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,665 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 71 02.12.2021 17:25:02,668 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:02,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:02,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,683 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:02,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:02,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:02,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:02,686 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132415, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:02,686 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132425, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:02,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:02,687 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:02,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,687 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:02,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:02,688 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59657ms for 60000ms of delay, sleeping for 353ms 02.12.2021 17:25:02,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,711 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:02,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:02,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:02,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:02,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,724 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:02,724 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:02,724 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=947) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:02,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:02,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,737 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,754 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:02,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,772 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,783 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:02,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:02,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:02,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:02,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,826 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=947): org.apache.kafka.common.requests.HeartbeatResponse@5c64b4be 02.12.2021 17:25:02,826 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:02,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:02,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,867 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:02,967 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:03,043 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58222ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:03,044 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:03,069 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:03,144 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:03,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,155 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,169 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:03,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,185 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:03,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,213 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:03,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:03,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:03,213 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:03,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,237 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,238 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,255 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,284 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:03,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:03,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:03,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:03,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,368 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,369 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,439 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:03,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:03,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:03,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:03,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:03,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,661 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:03,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,686 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,714 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:03,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:03,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:03,714 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,739 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,762 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,772 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:03,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,786 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:03,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:03,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:03,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:03,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:03,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,876 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,876 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,885 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=656): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:03,886 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:03,886 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:03,886 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:03,886 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=656): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:03,893 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:03,893 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:03,893 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:03,894 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:03,894 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=657) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,assignments=[]} 02.12.2021 17:25:03,894 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 380 02.12.2021 17:25:03,885 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1262): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0])]) 02.12.2021 17:25:03,893 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:03,895 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=657) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:03,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 367 02.12.2021 17:25:03,894 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0])]) 02.12.2021 17:25:03,902 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:03,902 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:03,902 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@661d7ab3, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7ebde251, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@53c01eb0, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4058d62a, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4d1de470, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5ca328e8, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@fc58852, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@92ae606, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2bb6bd37, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6e1724c1, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4db3a649, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@13f7732f, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@db63b63, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5ad03738, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@a0143c5, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@224737a1, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1e166da9, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4524e0a4, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@45bb8cae, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5eda4187, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@738c04d0, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@114ffbc5, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@808dbbe, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4d26541, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@17eb5b8f, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@79141b64, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@370fead3} 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Constructed client metadata {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, b340ff38-2a93-4102-8a87-c026f0ab26f7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 912326ee-6960-44d6-b637-f6bf4b4fb4d9=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7f1cad50-1794-4112-bb49-b2fba405b328=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 46d9f289-0630-4b19-b7f2-c59e02299906=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 8f66a158-1279-467f-a02e-ff9e9f38632b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 63f94986-6103-497f-970a-6fcbf19c32a5=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 941317db-575e-41d6-ace9-ee09236fae90=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:03,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:03,911 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:03,912 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:03,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:03,912 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Queueing Call(callName=metadata, deadlineMs=1638462363912, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:03,912 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:03,912 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462363912, tries=0, nextAllowedTryMs=0)] at 1638462303912 02.12.2021 17:25:03,912 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Metadata is ready to use. 02.12.2021 17:25:03,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_0, 0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:03,912 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Found least loaded node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) connected with no in-flight requests 02.12.2021 17:25:03,914 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Assigned Call(callName=metadata, deadlineMs=1638462363912, tries=0, nextAllowedTryMs=0) to node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_0, 0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client cfdd2ebd-a50b-40ed-9076-36cf63fe5b42 per-consumer assignment: 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client b340ff38-2a93-4102-8a87-c026f0ab26f7 per-consumer assignment: 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 912326ee-6960-44d6-b637-f6bf4b4fb4d9 per-consumer assignment: 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Removing task 0_8 from staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 due to tasks changing ownership. 02.12.2021 17:25:03,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested client 7f1cad50-1794-4112-bb49-b2fba405b328 to schedule a followup rebalance 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 7f1cad50-1794-4112-bb49-b2fba405b328 per-consumer assignment: 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Removing task 0_5 from staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:03,916 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d due to tasks changing ownership. 02.12.2021 17:25:03,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested client 46d9f289-0630-4b19-b7f2-c59e02299906 to schedule a followup rebalance 02.12.2021 17:25:03,917 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 46d9f289-0630-4b19-b7f2-c59e02299906 per-consumer assignment: 02.12.2021 17:25:03,917 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 8f66a158-1279-467f-a02e-ff9e9f38632b per-consumer assignment: 02.12.2021 17:25:03,920 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 63f94986-6103-497f-970a-6fcbf19c32a5 per-consumer assignment: 02.12.2021 17:25:03,929 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f per-consumer assignment: 02.12.2021 17:25:03,915 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-3.data.sta.redacted:9093 (id: 2 rack: null). correlationId=45 02.12.2021 17:25:03,929 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 941317db-575e-41d6-ace9-ee09236fae90 per-consumer assignment: 02.12.2021 17:25:03,934 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:03,934 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Finished assignment for group at generation 55: {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-0], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-2], userDataSize=48), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-7], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-3], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-1], userDataSize=48), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-6], userDataSize=48), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-4], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=Assignment(partitions=[], userDataSize=40)} 02.12.2021 17:25:03,935 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending leader SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 4, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:03,934 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=45) and timeout 30000 to node 2: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:03,939 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1263) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]}]} 02.12.2021 17:25:03,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 346 02.12.2021 17:25:03,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:03,939 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=60000) 02.12.2021 17:25:03,947 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:03,947 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462303947 02.12.2021 17:25:03,947 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59965) 02.12.2021 17:25:03,947 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Received METADATA response from node 2 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=45): org.apache.kafka.common.requests.MetadataResponse@7c74f0f4 02.12.2021 17:25:03,947 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:03,948 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Call(callName=metadata, deadlineMs=1638462363912, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:03,948 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462303947 02.12.2021 17:25:03,948 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=227135) 02.12.2021 17:25:03,960 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=657): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,960 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=657): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,960 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1263): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:03,962 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 55 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,963 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,963 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=55, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:03,963 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 55 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:03,965 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:03,965 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:03,963 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 55 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:03,966 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:03,966 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:03,966 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:03,963 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:03,966 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:03,966 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:03,966 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:03,966 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:03,966 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from RUNNING to REBALANCING 02.12.2021 17:25:03,966 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:03,966 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 324 02.12.2021 17:25:03,966 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:03,967 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:03,966 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:03,967 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:03,970 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:03,967 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:03,970 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 312 02.12.2021 17:25:03,970 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:03,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:03,976 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:04,043 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59222ms delay out of 60000ms. Remaining 778ms. Executor queue at 0 02.12.2021 17:25:04,046 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:04,047 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59077ms for 60000ms of delay, sleeping for 933ms 02.12.2021 17:25:04,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 218 02.12.2021 17:25:04,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 201 02.12.2021 17:25:04,074 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:04,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:04,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:04,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,163 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,163 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 116 02.12.2021 17:25:04,176 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:04,186 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 76 02.12.2021 17:25:04,186 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:04,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,191 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,216 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:04,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:04,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:04,216 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,240 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:04,262 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:04,263 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:04,263 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:04,263 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:04,263 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:04,263 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:04,263 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:04,265 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:04,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:04,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,265 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 16 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:04,275 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,286 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,286 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:04,286 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:04,286 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:04,286 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:04,286 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:04,286 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:04,287 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:04,287 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from REBALANCING to RUNNING 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:04,288 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:04,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:04,288 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 55 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:04,288 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:04,288 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:04,288 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:04,288 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1264) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:04,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:04,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:04,370 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:04,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:04,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:04,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,381 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,381 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,440 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:04,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:04,479 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:04,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:04,541 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:04,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:04,580 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:04,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:04,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:04,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,666 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:04,681 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:04,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:04,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,692 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,692 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,718 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:04,718 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,718 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:04,718 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,718 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,718 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Committing all active tasks [] and standby tasks [] since 30238ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:04,718 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:04,718 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:04,718 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,742 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,766 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,766 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,767 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,776 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,771 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:04,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,789 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:04,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:04,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:04,789 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:04,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:04,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 397 02.12.2021 17:25:04,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:04,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,889 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 387 02.12.2021 17:25:04,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:04,981 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:04,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:04,998 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 284 02.12.2021 17:25:05,014 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59763ms for 60000ms of delay, sleeping for 247ms 02.12.2021 17:25:05,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:05,058 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=664): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:05,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:05,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:05,066 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:05,058 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=688): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:05,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:05,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:05,067 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:05,067 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:05,067 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=689) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:05,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:05,058 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1027): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:05,067 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:05,068 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:05,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=665) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,assignments=[]} 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:05,070 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1028) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,assignments=[]} 02.12.2021 17:25:05,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 122 02.12.2021 17:25:05,071 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 147 02.12.2021 17:25:05,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:05,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:05,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 177 02.12.2021 17:25:05,106 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1028): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:05,106 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:05,106 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:05,106 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=665): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,107 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,107 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:05,107 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 186 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:05,107 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:05,107 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:05,107 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:05,107 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:05,107 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:05,107 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from RUNNING to REBALANCING 02.12.2021 17:25:05,107 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:05,107 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 85 02.12.2021 17:25:05,107 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 186 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:05,107 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=689): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,108 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:05,108 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:05,108 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,108 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=186, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:05,108 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:05,108 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 186 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:05,108 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:05,108 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:05,108 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:05,108 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:05,108 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:05,108 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:05,109 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:05,109 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:05,109 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:05,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 112 02.12.2021 17:25:05,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 58 02.12.2021 17:25:05,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:05,167 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:05,168 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:05,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:05,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,168 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,168 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:05,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:05,193 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:05,193 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:05,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:05,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,194 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,194 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,205 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 77 02.12.2021 17:25:05,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 9 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:05,220 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:05,220 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from REBALANCING to RUNNING 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Committing all active tasks [] and standby tasks [] since 30740ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:05,220 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:05,220 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 186 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:05,221 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:05,221 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:05,221 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:05,221 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1029) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:05,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:05,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,243 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,243 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 20 02.12.2021 17:25:05,262 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59388ms for 60000ms of delay, sleeping for 622ms 02.12.2021 17:25:05,263 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:05,263 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:05,263 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=650) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:05,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 480 02.12.2021 17:25:05,264 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=650): org.apache.kafka.common.requests.HeartbeatResponse@e7eac7d 02.12.2021 17:25:05,264 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:05,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 479 02.12.2021 17:25:05,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 479 02.12.2021 17:25:05,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:05,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:05,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,277 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,284 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:05,284 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:05,284 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=650) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:05,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 111 02.12.2021 17:25:05,286 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=650): org.apache.kafka.common.requests.HeartbeatResponse@6bbdfe7 02.12.2021 17:25:05,286 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:05,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 109 02.12.2021 17:25:05,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 109 02.12.2021 17:25:05,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,290 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:05,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:05,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:05,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:05,364 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 379 02.12.2021 17:25:05,368 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:05,379 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:05,379 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:05,386 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 9 02.12.2021 17:25:05,394 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:05,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:05,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,403 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,403 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,465 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 278 02.12.2021 17:25:05,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:05,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:05,485 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 293 02.12.2021 17:25:05,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:05,503 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:05,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 176 02.12.2021 17:25:05,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:05,581 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:05,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:05,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:05,606 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:05,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 76 02.12.2021 17:25:05,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:05,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,674 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,679 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:05,689 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 89 02.12.2021 17:25:05,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,694 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:05,721 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,733 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:05,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:05,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:05,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,733 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:05,737 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:05,737 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=948) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:05,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:05,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,746 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,746 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,779 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,780 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:05,791 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,791 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,791 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:05,791 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,791 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:05,791 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,791 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,791 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:05,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:05,794 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:05,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:05,838 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=948): org.apache.kafka.common.requests.HeartbeatResponse@2fc73a75 02.12.2021 17:25:05,841 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:05,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:05,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:05,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:05,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:05,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:05,898 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59601ms for 60000ms of delay, sleeping for 409ms 02.12.2021 17:25:05,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:05,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,916 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:05,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:05,982 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:05,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:05,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 287 02.12.2021 17:25:06,020 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:06,053 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:06,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:06,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:06,102 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:06,104 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 178 02.12.2021 17:25:06,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:06,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:06,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:06,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:06,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,185 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,196 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,196 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 73 02.12.2021 17:25:06,220 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:06,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,234 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:06,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:06,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:06,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:06,235 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,246 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,247 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,280 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,280 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,280 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,280 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,281 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,282 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:06,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:06,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,297 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:06,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:06,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:06,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:06,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:06,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,309 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59132ms for 60000ms of delay, sleeping for 878ms 02.12.2021 17:25:06,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:06,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:06,383 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:06,383 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:06,389 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:06,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:06,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:06,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:06,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:06,483 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:06,490 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:06,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:06,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:06,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:06,584 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:06,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:06,591 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:06,600 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:06,620 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:06,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:06,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:06,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:06,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,686 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,687 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,697 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,697 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,720 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:06,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:06,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,736 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:06,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:06,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:06,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:06,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,749 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 111 02.12.2021 17:25:06,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,783 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 103 02.12.2021 17:25:06,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:06,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,798 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:06,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:06,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:06,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:06,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,797 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:06,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:06,852 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:06,887 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:06,884 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 2 02.12.2021 17:25:06,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 10 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:06,904 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:06,912 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 55 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,912 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=658) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:06,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 371 02.12.2021 17:25:06,911 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:06,914 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 55 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,914 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=658) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:06,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,914 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 368 02.12.2021 17:25:06,914 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=658): org.apache.kafka.common.requests.HeartbeatResponse@875ce61 02.12.2021 17:25:06,912 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=658): org.apache.kafka.common.requests.HeartbeatResponse@74edee21 02.12.2021 17:25:06,916 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:06,921 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 55 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:06,921 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:06,921 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:06,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,917 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:06,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,921 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,924 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=659) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:06,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 358 02.12.2021 17:25:06,924 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,923 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 55 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:06,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,929 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:06,939 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:06,939 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,943 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=659) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:06,950 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 333 02.12.2021 17:25:06,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:06,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 192 02.12.2021 17:25:07,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 192 02.12.2021 17:25:07,012 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:07,014 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:07,039 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 391 02.12.2021 17:25:07,053 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:07,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:07,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:07,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:07,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:07,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:07,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,186 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,188 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59360ms for 60000ms of delay, sleeping for 650ms 02.12.2021 17:25:07,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,198 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,198 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:07,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,241 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:07,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:07,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:07,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,250 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:07,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,299 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:07,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:07,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:07,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:07,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:07,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:07,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:07,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,433 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,434 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:07,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:07,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:07,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:07,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:07,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:07,601 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:07,635 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:07,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:07,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:07,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,694 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 373 02.12.2021 17:25:07,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,706 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,706 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 360 02.12.2021 17:25:07,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:07,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,742 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:07,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:07,743 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:07,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,750 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:07,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,784 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 272 02.12.2021 17:25:07,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,807 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:07,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:07,808 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 259 02.12.2021 17:25:07,840 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:07,841 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132425, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:07,841 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132435, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:07,841 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:07,845 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59064ms for 60000ms of delay, sleeping for 946ms 02.12.2021 17:25:07,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:07,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 171 02.12.2021 17:25:07,913 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 153 02.12.2021 17:25:07,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,939 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,940 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 344 02.12.2021 17:25:07,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:07,998 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 69 02.12.2021 17:25:08,013 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 53 02.12.2021 17:25:08,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 244 02.12.2021 17:25:08,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:08,067 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,068 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 186 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,068 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=666) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null} 02.12.2021 17:25:08,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:08,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 138 02.12.2021 17:25:08,072 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,072 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 186 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,072 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:08,070 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=666): org.apache.kafka.common.requests.HeartbeatResponse@2c666518 02.12.2021 17:25:08,073 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:08,073 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 186 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:08,072 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=690) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null} 02.12.2021 17:25:08,075 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:08,076 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:08,076 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,075 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 119 02.12.2021 17:25:08,078 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=690): org.apache.kafka.common.requests.HeartbeatResponse@320e8fcf 02.12.2021 17:25:08,077 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=667) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:08,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 125 02.12.2021 17:25:08,081 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:08,082 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 186 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:08,083 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:08,083 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:08,083 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,086 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=691) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:08,087 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 107 02.12.2021 17:25:08,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 135 02.12.2021 17:25:08,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:08,168 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:08,175 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:08,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,195 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,208 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,208 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:08,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,244 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:08,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:08,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:08,244 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,250 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 13 02.12.2021 17:25:08,251 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 33 02.12.2021 17:25:08,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,264 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,264 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:08,264 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=651) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:08,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 486 02.12.2021 17:25:08,264 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=651): org.apache.kafka.common.requests.HeartbeatResponse@30b26d3 02.12.2021 17:25:08,265 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:08,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 485 02.12.2021 17:25:08,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 485 02.12.2021 17:25:08,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,289 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:08,289 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=651) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:08,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 151 02.12.2021 17:25:08,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,290 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=651): org.apache.kafka.common.requests.HeartbeatResponse@42faa53f 02.12.2021 17:25:08,291 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:08,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 149 02.12.2021 17:25:08,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 149 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:08,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 385 02.12.2021 17:25:08,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 48 02.12.2021 17:25:08,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,441 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,441 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:08,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:08,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 180 02.12.2021 17:25:08,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:08,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:08,696 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,696 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,697 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,697 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,710 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:08,750 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=949) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:08,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:08,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,792 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,792 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,793 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,793 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,794 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59015ms for 60000ms of delay, sleeping for 995ms 02.12.2021 17:25:08,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,810 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:08,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:08,811 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,811 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,811 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:08,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:08,851 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=949): org.apache.kafka.common.requests.HeartbeatResponse@53a650ce 02.12.2021 17:25:08,851 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:08,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:08,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,943 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:08,949 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 392 02.12.2021 17:25:09,056 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:09,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:09,160 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:09,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:09,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,204 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:09,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,213 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,213 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,251 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:09,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,258 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:09,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,293 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,312 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:09,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:09,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:09,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:09,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:09,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:09,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:09,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,457 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,457 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:09,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:09,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:09,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:09,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,705 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,706 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,715 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,717 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,754 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:09,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:09,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:09,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:09,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,761 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,761 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,763 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:09,790 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59928ms for 60000ms of delay, sleeping for 82ms 02.12.2021 17:25:09,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,795 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:09,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,800 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,800 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,813 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:09,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:09,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:09,814 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:09,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:09,879 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59741ms for 60000ms of delay, sleeping for 269ms 02.12.2021 17:25:09,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:09,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,965 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:10,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 395 02.12.2021 17:25:10,152 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59988ms for 60000ms of delay, sleeping for 22ms 02.12.2021 17:25:10,164 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:10,175 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59673ms for 60000ms of delay, sleeping for 337ms 02.12.2021 17:25:10,198 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:10,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,206 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,207 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:10,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:10,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,223 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,223 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,255 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:10,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:10,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:10,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:10,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,262 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30166ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:10,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30189ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:10,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,315 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:10,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:10,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:10,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:10,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:10,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:10,463 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:10,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 517 ms and fetched 0 records 02.12.2021 17:25:10,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,496 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,515 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59451ms for 60000ms of delay, sleeping for 559ms 02.12.2021 17:25:10,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:10,597 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:10,664 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:10,697 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:10,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,706 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:10,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,727 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,758 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:10,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:10,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:10,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:10,758 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,762 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,763 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30667ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:10,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:10,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,803 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30691ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:10,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:10,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,816 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:10,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:10,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:10,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:10,817 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:10,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:10,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:11,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:11,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,005 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,008 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 279 02.12.2021 17:25:11,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:11,075 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59964ms for 60000ms of delay, sleeping for 46ms 02.12.2021 17:25:11,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 177 02.12.2021 17:25:11,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:11,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,124 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132435, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:11,124 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132445, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:11,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:11,125 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:11,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,125 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:11,126 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59967ms for 60000ms of delay, sleeping for 43ms 02.12.2021 17:25:11,168 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:11,171 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59607ms for 60000ms of delay, sleeping for 403ms 02.12.2021 17:25:11,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,208 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,208 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 77 02.12.2021 17:25:11,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,229 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,260 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:11,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,263 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,264 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:11,264 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=652) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:11,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:11,265 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=652): org.apache.kafka.common.requests.HeartbeatResponse@2cb533dc 02.12.2021 17:25:11,265 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:11,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 499 02.12.2021 17:25:11,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 499 02.12.2021 17:25:11,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,288 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:11,288 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:11,288 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=652) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:11,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 220 02.12.2021 17:25:11,288 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=652): org.apache.kafka.common.requests.HeartbeatResponse@356163fb 02.12.2021 17:25:11,289 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:11,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 219 02.12.2021 17:25:11,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 219 02.12.2021 17:25:11,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:11,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,300 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31170ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31193ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,318 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:11,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:11,389 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 119 02.12.2021 17:25:11,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:11,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 15 02.12.2021 17:25:11,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,511 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,512 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,512 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:11,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:11,575 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59500ms for 60000ms of delay, sleeping for 510ms 02.12.2021 17:25:11,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:11,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:11,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,714 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,714 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:11,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30257ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,717 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30186ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,761 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:11,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,758 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:11,765 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:11,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,768 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:11,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,775 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:11,768 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=950) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:11,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,806 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31678ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,807 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31700ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,813 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,819 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:11,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:11,813 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:11,876 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=950): org.apache.kafka.common.requests.HeartbeatResponse@587c18 02.12.2021 17:25:11,878 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:11,925 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:11,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:12,013 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,020 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,020 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,023 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,087 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:12,103 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59428ms for 60000ms of delay, sleeping for 582ms 02.12.2021 17:25:12,124 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:12,201 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30761ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,218 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30687ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,231 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:12,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,275 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 518 ms and fetched 0 records 02.12.2021 17:25:12,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,279 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:12,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:12,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,280 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,280 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:12,280 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32201ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,320 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:12,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:12,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:12,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 518 ms and fetched 0 records 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32200ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,330 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:12,430 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:12,478 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:12,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:12,625 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:12,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:12,687 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59611ms for 60000ms of delay, sleeping for 399ms 02.12.2021 17:25:12,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31263ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,720 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31188ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,732 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,776 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,783 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:12,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:12,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:12,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32702ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,815 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,821 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:12,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:12,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:12,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32701ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,831 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:12,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:12,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:13,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,025 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,077 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:13,087 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59941ms for 60000ms of delay, sleeping for 69ms 02.12.2021 17:25:13,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:13,159 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59417ms for 60000ms of delay, sleeping for 593ms 02.12.2021 17:25:13,178 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:13,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:13,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,222 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31766ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,223 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,223 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31689ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,276 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,276 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,276 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,276 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,276 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,276 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,276 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,276 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:13,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,288 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:13,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:13,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:13,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:13,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33203ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,322 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:13,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:13,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:13,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:13,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,327 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33202ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,377 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:13,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:13,477 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:13,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:13,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:13,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:13,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,725 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,726 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:13,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32275ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,734 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,734 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,735 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32191ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,755 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59701ms for 60000ms of delay, sleeping for 309ms 02.12.2021 17:25:13,783 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:13,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,785 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 478 02.12.2021 17:25:13,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:13,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,792 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:13,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:13,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:13,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:13,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33704ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,817 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,823 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:13,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:13,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:13,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:13,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,833 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:13,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,833 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33704ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,834 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,888 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 376 02.12.2021 17:25:13,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:13,930 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1029): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:13,936 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:13,937 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1030) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,assignments=[]} 02.12.2021 17:25:13,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 359 02.12.2021 17:25:13,929 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=667): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:13,929 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=691): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:13,940 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:13,941 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:13,942 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:13,940 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:13,942 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:13,942 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:13,942 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:13,942 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=692) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=668) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,assignments=[]} 02.12.2021 17:25:13,948 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 291 02.12.2021 17:25:13,948 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 286 02.12.2021 17:25:13,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 275 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,037 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 251 02.12.2021 17:25:14,061 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=668): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,061 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=692): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,061 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,062 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:14,061 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1030): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:14,062 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:14,063 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:14,063 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 187 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:14,062 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 187 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:14,064 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:14,064 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,064 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,064 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:14,064 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,064 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from RUNNING to REBALANCING 02.12.2021 17:25:14,064 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:14,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 178 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,065 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132445, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:14,065 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132455, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:14,065 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:14,066 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59950ms for 60000ms of delay, sleeping for 60ms 02.12.2021 17:25:14,062 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,066 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=187, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:14,066 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 187 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:14,066 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:14,066 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,066 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,066 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:14,066 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,066 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:14,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 172 02.12.2021 17:25:14,063 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:14,070 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,070 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:14,071 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:14,075 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,075 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:14,075 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 234 02.12.2021 17:25:14,089 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 175 02.12.2021 17:25:14,108 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=659): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:14,108 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:14,114 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:14,108 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=659): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:14,107 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1264): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0])]) 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 28, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0])]) 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:14,115 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@20e0691e, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6d9a69ae, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@348c8042, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7c1dad72, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5f168376, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@61ff4623, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@312154ce, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@204619a5, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4ba023e8, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@50f5aae1, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6b490315, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7bd2b4e8, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@31a76719, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4cf8716e, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6f8a824f, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1110ea2, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@b042e36, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4f85d39, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3f5b2f13, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@36364244, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1c69bf2a, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5a3e280b, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3c7ba7e8, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@385bf641, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@691a9e8f, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@424f9dd7, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@236a3691} 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Constructed client metadata {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, b340ff38-2a93-4102-8a87-c026f0ab26f7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 912326ee-6960-44d6-b637-f6bf4b4fb4d9=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7f1cad50-1794-4112-bb49-b2fba405b328=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 46d9f289-0630-4b19-b7f2-c59e02299906=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 8f66a158-1279-467f-a02e-ff9e9f38632b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 63f94986-6103-497f-970a-6fcbf19c32a5=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 941317db-575e-41d6-ace9-ee09236fae90=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:14,117 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:14,117 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:14,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:14,117 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:14,114 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:14,117 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:14,117 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:14,114 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:14,118 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:14,118 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:14,117 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:14,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:14,118 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Queueing Call(callName=metadata, deadlineMs=1638462374118, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:14,118 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:14,119 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462374118, tries=0, nextAllowedTryMs=0)] at 1638462314119 02.12.2021 17:25:14,119 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Metadata is ready to use. 02.12.2021 17:25:14,118 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=660) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,assignments=[]} 02.12.2021 17:25:14,118 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=660) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:14,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 215 02.12.2021 17:25:14,119 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Found least loaded node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) connected with no in-flight requests 02.12.2021 17:25:14,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_0, 0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:14,120 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Assigned Call(callName=metadata, deadlineMs=1638462374118, tries=0, nextAllowedTryMs=0) to node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) 02.12.2021 17:25:14,120 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-3.data.sta.redacted:9093 (id: 2 rack: null). correlationId=46 02.12.2021 17:25:14,120 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=46) and timeout 30000 to node 2: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:14,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:14,120 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59999) 02.12.2021 17:25:14,120 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_0, 0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:14,120 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:14,120 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:14,121 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client cfdd2ebd-a50b-40ed-9076-36cf63fe5b42 per-consumer assignment: 02.12.2021 17:25:14,120 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462314120 02.12.2021 17:25:14,121 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59998) 02.12.2021 17:25:14,121 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client b340ff38-2a93-4102-8a87-c026f0ab26f7 per-consumer assignment: 02.12.2021 17:25:14,121 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 912326ee-6960-44d6-b637-f6bf4b4fb4d9 per-consumer assignment: 02.12.2021 17:25:14,121 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Received METADATA response from node 2 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=46): org.apache.kafka.common.requests.MetadataResponse@3d2b26e1 02.12.2021 17:25:14,122 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:14,121 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Removing task 0_8 from staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:14,122 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 due to tasks changing ownership. 02.12.2021 17:25:14,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested client 7f1cad50-1794-4112-bb49-b2fba405b328 to schedule a followup rebalance 02.12.2021 17:25:14,122 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Call(callName=metadata, deadlineMs=1638462374118, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:14,122 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462314122 02.12.2021 17:25:14,122 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=216960) 02.12.2021 17:25:14,122 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 7f1cad50-1794-4112-bb49-b2fba405b328 per-consumer assignment: 02.12.2021 17:25:14,123 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Removing task 0_5 from staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:14,123 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d due to tasks changing ownership. 02.12.2021 17:25:14,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested client 46d9f289-0630-4b19-b7f2-c59e02299906 to schedule a followup rebalance 02.12.2021 17:25:14,127 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 46d9f289-0630-4b19-b7f2-c59e02299906 per-consumer assignment: 02.12.2021 17:25:14,127 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 8f66a158-1279-467f-a02e-ff9e9f38632b per-consumer assignment: 02.12.2021 17:25:14,127 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 63f94986-6103-497f-970a-6fcbf19c32a5 per-consumer assignment: 02.12.2021 17:25:14,127 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f per-consumer assignment: 02.12.2021 17:25:14,127 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 941317db-575e-41d6-ace9-ee09236fae90 per-consumer assignment: 02.12.2021 17:25:14,127 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:14,128 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Finished assignment for group at generation 56: {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-0], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-2], userDataSize=48), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-7], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-3], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-1], userDataSize=48), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-6], userDataSize=48), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-4], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=Assignment(partitions=[], userDataSize=40)} 02.12.2021 17:25:14,128 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending leader SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 4, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:14,128 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1265) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]}]} 02.12.2021 17:25:14,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:14,131 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59330ms for 60000ms of delay, sleeping for 680ms 02.12.2021 17:25:14,138 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 150 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=660): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,151 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:14,151 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 56 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:14,151 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:14,151 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,151 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,151 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:14,151 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=660): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1265): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:14,151 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,151 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from RUNNING to REBALANCING 02.12.2021 17:25:14,151 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:14,152 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 167 02.12.2021 17:25:14,151 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:14,151 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:14,152 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=56, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:14,152 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 56 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:14,152 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:14,152 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,152 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,153 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:14,153 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,153 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:14,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 183 02.12.2021 17:25:14,152 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 56 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:14,153 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:14,153 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,153 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:14,153 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:14,153 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,153 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:14,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 174 02.12.2021 17:25:14,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 74 02.12.2021 17:25:14,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 67 02.12.2021 17:25:14,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 113 02.12.2021 17:25:14,191 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 73 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,234 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32777ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,235 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,238 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 50 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,239 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,239 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32696ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 65 02.12.2021 17:25:14,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 81 02.12.2021 17:25:14,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 70 02.12.2021 17:25:14,264 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:14,264 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:14,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,264 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=653) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:14,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 21 02.12.2021 17:25:14,268 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=653): org.apache.kafka.common.requests.HeartbeatResponse@4f35f306 02.12.2021 17:25:14,268 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:14,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 18 02.12.2021 17:25:14,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 18 02.12.2021 17:25:14,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 12 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,288 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:14,288 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:14,289 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=653) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:14,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 249 02.12.2021 17:25:14,289 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=653): org.apache.kafka.common.requests.HeartbeatResponse@23a5aeae 02.12.2021 17:25:14,289 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:14,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 248 02.12.2021 17:25:14,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 248 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,296 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,296 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from REBALANCING to RUNNING 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,296 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:14,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,297 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 187 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:14,297 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:14,297 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:14,297 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:14,297 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1031) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:14,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:14,317 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,317 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,317 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,318 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 34205ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:14,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,325 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:14,325 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,325 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,325 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:14,325 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,326 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,326 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 56 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:14,326 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:14,326 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1266) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:14,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,334 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:14,334 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,334 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,334 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,335 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,335 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from REBALANCING to RUNNING 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 34205ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:14,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:14,355 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:14,384 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:14,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 148 02.12.2021 17:25:14,389 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:14,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:14,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:14,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:14,440 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:14,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 45 02.12.2021 17:25:14,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:14,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:14,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:14,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:14,537 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,541 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:14,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:14,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:14,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:14,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:14,638 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:14,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:14,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:14,720 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:14,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,736 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,736 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:14,740 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:14,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,741 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,789 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,797 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:14,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,797 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,798 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:14,798 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:14,798 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=951) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:14,814 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59733ms for 60000ms of delay, sleeping for 277ms 02.12.2021 17:25:14,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,818 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,818 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,827 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:14,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,836 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:14,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:14,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:14,891 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:14,899 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=951): org.apache.kafka.common.requests.HeartbeatResponse@79e0f9e9 02.12.2021 17:25:14,901 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:14,919 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:14,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:14,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:14,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:14,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:15,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:15,023 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:15,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:15,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,038 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,039 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:15,042 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:15,093 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58896ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:15,101 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:15,123 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:15,137 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:15,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:15,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 194 02.12.2021 17:25:15,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:15,201 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:15,224 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:15,236 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,237 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,237 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,239 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:15,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,241 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:15,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,289 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,298 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:15,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:15,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:15,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:15,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,319 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,328 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:15,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:15,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:15,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:15,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,336 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:15,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:15,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:15,390 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:15,420 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:15,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:15,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:15,440 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:15,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:15,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:15,521 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:15,537 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:15,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:15,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,539 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,540 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:15,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:15,621 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:15,637 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:15,638 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:15,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:15,644 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:15,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:15,722 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:15,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:15,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,740 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:15,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:15,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,745 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:15,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,748 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,748 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,799 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,800 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:15,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:15,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:15,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:15,802 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:15,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,806 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,824 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,833 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:15,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:15,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:15,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:15,834 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:15,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:15,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,839 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,843 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:15,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:15,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:15,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:15,926 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:15,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:15,949 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:15,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:15,949 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:16,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:16,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:16,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:16,049 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:16,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:16,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,053 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 189 02.12.2021 17:25:16,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,093 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59896ms delay out of 60000ms. Remaining 104ms. Executor queue at 0 02.12.2021 17:25:16,095 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59749ms for 60000ms of delay, sleeping for 261ms 02.12.2021 17:25:16,114 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:16,129 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:16,140 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:16,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:16,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:16,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:16,214 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:16,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:16,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:16,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,245 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,245 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,248 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:16,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,302 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:16,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:16,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,303 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:16,303 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:16,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,308 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:16,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,326 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,334 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:16,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:16,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:16,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:16,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,340 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:16,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:16,357 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59557ms for 60000ms of delay, sleeping for 453ms 02.12.2021 17:25:16,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:16,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:16,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,440 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,447 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:16,449 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:16,459 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:16,508 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:16,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:16,541 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:16,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:16,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:16,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:16,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,560 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:16,628 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:16,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:16,648 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:16,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:16,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 394 02.12.2021 17:25:16,714 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:16,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:16,744 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:16,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,746 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,746 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:16,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,749 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 191 02.12.2021 17:25:16,768 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:16,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,804 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:16,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:16,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:16,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:16,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:16,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,812 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 452 02.12.2021 17:25:16,813 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59669ms for 60000ms of delay, sleeping for 341ms 02.12.2021 17:25:16,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,828 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,828 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 286 02.12.2021 17:25:16,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,835 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:16,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:16,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:16,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:16,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:16,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,842 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,842 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 275 02.12.2021 17:25:16,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:16,852 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 90 02.12.2021 17:25:16,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:16,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 352 02.12.2021 17:25:16,928 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 186 02.12.2021 17:25:16,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 175 02.12.2021 17:25:16,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,943 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:16,943 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 187 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:16,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,943 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:16,944 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 187 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:16,944 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=669) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null} 02.12.2021 17:25:16,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 308 02.12.2021 17:25:16,945 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=669): org.apache.kafka.common.requests.HeartbeatResponse@7b3336fe 02.12.2021 17:25:16,945 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:16,946 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 187 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:16,946 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:16,946 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:16,946 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:16,946 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=670) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:16,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 305 02.12.2021 17:25:16,944 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=693) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null} 02.12.2021 17:25:16,947 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=693): org.apache.kafka.common.requests.HeartbeatResponse@7612d090 02.12.2021 17:25:16,947 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:16,957 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 187 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:16,957 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:16,957 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:16,957 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:16,957 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=694) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:16,957 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 289 02.12.2021 17:25:16,969 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:17,016 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 248 02.12.2021 17:25:17,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 86 02.12.2021 17:25:17,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 74 02.12.2021 17:25:17,044 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:17,047 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 222 02.12.2021 17:25:17,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,116 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,116 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 56 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,116 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=661) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:17,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 212 02.12.2021 17:25:17,117 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=661): org.apache.kafka.common.requests.HeartbeatResponse@51dc502f 02.12.2021 17:25:17,117 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:17,117 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 56 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:17,117 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:17,117 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:17,117 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,117 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=662) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:17,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 211 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 146 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,118 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 56 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,118 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=661) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:17,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 223 02.12.2021 17:25:17,119 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=661): org.apache.kafka.common.requests.HeartbeatResponse@6115d14 02.12.2021 17:25:17,119 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:17,119 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 56 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:17,119 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:17,119 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:17,119 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,119 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=662) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:17,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 222 02.12.2021 17:25:17,156 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59749ms for 60000ms of delay, sleeping for 261ms 02.12.2021 17:25:17,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 122 02.12.2021 17:25:17,217 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:17,218 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 46 02.12.2021 17:25:17,219 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:17,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,247 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,247 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,252 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,265 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,265 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:17,265 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=654) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:17,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 47 02.12.2021 17:25:17,266 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=654): org.apache.kafka.common.requests.HeartbeatResponse@34d3b3d9 02.12.2021 17:25:17,266 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:17,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 46 02.12.2021 17:25:17,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 46 02.12.2021 17:25:17,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 22 02.12.2021 17:25:17,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,289 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,289 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:17,289 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=654) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:17,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 277 02.12.2021 17:25:17,290 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=654): org.apache.kafka.common.requests.HeartbeatResponse@7bb909da 02.12.2021 17:25:17,290 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:17,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 276 02.12.2021 17:25:17,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 276 02.12.2021 17:25:17,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,310 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:17,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,313 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:17,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,329 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,336 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:17,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 176 02.12.2021 17:25:17,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:17,418 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59463ms for 60000ms of delay, sleeping for 547ms 02.12.2021 17:25:17,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:17,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:17,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:17,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,573 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:17,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:17,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:17,717 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:17,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:17,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,752 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,753 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:17,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,754 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:17,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:17,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,812 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:17,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,813 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,814 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:17,813 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,814 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:17,814 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=952) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:17,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:17,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,835 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:17,914 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:17,914 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=952): org.apache.kafka.common.requests.HeartbeatResponse@6f627032 02.12.2021 17:25:17,914 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:17,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:17,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,968 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132455, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:17,968 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132465, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:17,968 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:17,969 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:17,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,969 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:17,969 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59702ms for 60000ms of delay, sleeping for 308ms 02.12.2021 17:25:17,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:18,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:18,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,076 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:18,177 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:18,216 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:18,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,253 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,254 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,255 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:18,278 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59623ms for 60000ms of delay, sleeping for 387ms 02.12.2021 17:25:18,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:18,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,313 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:18,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:18,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:18,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:18,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,314 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:18,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,340 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,346 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:18,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,349 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:18,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:18,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:18,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:18,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,377 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:18,415 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:18,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:18,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:18,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,617 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:18,669 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59818ms for 60000ms of delay, sleeping for 192ms 02.12.2021 17:25:18,683 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:18,718 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:18,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,754 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,756 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,757 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,785 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:18,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,814 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:18,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:18,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:18,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:18,814 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:18,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,819 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,820 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,842 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,842 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,847 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,848 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,848 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,848 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:18,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,851 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:18,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:18,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:18,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:18,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,862 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58645ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:18,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:18,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:18,986 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:19,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 287 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,078 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:19,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:19,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:19,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,255 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,256 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,256 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,256 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,258 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,258 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,258 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,256 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,281 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:19,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,315 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:19,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:19,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:19,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:19,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,320 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,343 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:19,357 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,357 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,382 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:19,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:19,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:19,521 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:19,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:19,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,580 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,622 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:19,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:19,729 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:19,768 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:19,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,772 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,768 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:19,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,773 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,784 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:19,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,818 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:19,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:19,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:19,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:19,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,821 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,820 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,822 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 443 02.12.2021 17:25:19,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,844 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:19,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,858 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,859 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,859 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:19,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:19,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:19,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:19,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,862 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59645ms delay out of 60000ms. Remaining 355ms. Executor queue at 0 02.12.2021 17:25:19,863 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59101ms for 60000ms of delay, sleeping for 909ms 02.12.2021 17:25:19,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:19,923 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 342 02.12.2021 17:25:19,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:20,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 232 02.12.2021 17:25:20,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:20,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,084 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,087 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 202 02.12.2021 17:25:20,135 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 130 02.12.2021 17:25:20,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 102 02.12.2021 17:25:20,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 25 02.12.2021 17:25:20,266 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:20,266 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:20,266 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=655) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:20,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:20,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 55 02.12.2021 17:25:20,268 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=655): org.apache.kafka.common.requests.HeartbeatResponse@6e34362e 02.12.2021 17:25:20,268 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:20,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 54 02.12.2021 17:25:20,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 53 02.12.2021 17:25:20,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,273 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,274 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 2 02.12.2021 17:25:20,290 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:20,290 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:20,290 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=655) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:20,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:20,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:20,291 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=655): org.apache.kafka.common.requests.HeartbeatResponse@6cfd88cf 02.12.2021 17:25:20,291 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:20,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:20,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:20,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:20,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,345 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,359 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,362 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:20,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,362 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:20,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:20,391 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:20,423 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:20,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:20,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:20,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:20,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,589 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:20,625 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:20,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:20,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:20,774 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59961ms for 60000ms of delay, sleeping for 49ms 02.12.2021 17:25:20,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:20,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,775 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,775 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,776 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:20,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,778 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:20,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:20,821 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:20,821 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:20,821 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=953) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:20,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,823 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:20,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:20,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:20,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:20,823 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,825 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:20,827 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59771ms for 60000ms of delay, sleeping for 239ms 02.12.2021 17:25:20,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,846 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,847 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:20,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:20,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,891 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:20,922 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=953): org.apache.kafka.common.requests.HeartbeatResponse@ea36d30 02.12.2021 17:25:20,922 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:20,925 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:20,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:21,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:21,068 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59799ms for 60000ms of delay, sleeping for 211ms 02.12.2021 17:25:21,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:21,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,090 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:21,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:21,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,277 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,278 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,280 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59216ms for 60000ms of delay, sleeping for 794ms 02.12.2021 17:25:21,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:21,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,324 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:21,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:21,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,328 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,347 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:21,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:21,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:21,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,591 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,591 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:21,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:21,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,778 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,778 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,780 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:21,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,825 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:21,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:21,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,832 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,832 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,848 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,848 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,849 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:21,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,874 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:21,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,902 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:21,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:21,954 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1266): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0])]) 02.12.2021 17:25:21,955 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=662): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:21,955 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:21,955 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:21,955 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=663) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,assignments=[]} 02.12.2021 17:25:21,955 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=662): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:21,956 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=663) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:21,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 394 02.12.2021 17:25:21,955 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 29, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0])]) 02.12.2021 17:25:21,957 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:21,957 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:21,957 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@44e42543, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@79b725b6, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4728be72, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3584c6b8, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@502f2376, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@10f53650, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@12acd112, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@18e64f9, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2813d32d, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@116d27c9, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@653d40b7, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@17664945, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@215a6a3f, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@597152b2, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@56c699d3, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@734fce2f, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1b659eb, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@52816fad, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@306b6c2f, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3d58f2bb, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@e129dfa, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@57b77792, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3c56735f, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@46e19344, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5c062b2f, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3061ccda, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3f228d02} 02.12.2021 17:25:21,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 412 02.12.2021 17:25:21,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:21,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Constructed client metadata {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, b340ff38-2a93-4102-8a87-c026f0ab26f7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 912326ee-6960-44d6-b637-f6bf4b4fb4d9=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7f1cad50-1794-4112-bb49-b2fba405b328=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 46d9f289-0630-4b19-b7f2-c59e02299906=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 8f66a158-1279-467f-a02e-ff9e9f38632b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 63f94986-6103-497f-970a-6fcbf19c32a5=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 941317db-575e-41d6-ace9-ee09236fae90=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:22,002 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Queueing Call(callName=metadata, deadlineMs=1638462382000, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:22,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_0, 0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:22,005 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:22,028 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462382000, tries=0, nextAllowedTryMs=0)] at 1638462322028 02.12.2021 17:25:22,028 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Metadata is ready to use. 02.12.2021 17:25:22,028 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Found least loaded node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) connected with no in-flight requests 02.12.2021 17:25:22,028 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Assigned Call(callName=metadata, deadlineMs=1638462382000, tries=0, nextAllowedTryMs=0) to node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) 02.12.2021 17:25:22,005 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_0, 0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:22,028 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-3.data.sta.redacted:9093 (id: 2 rack: null). correlationId=47 02.12.2021 17:25:22,028 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:22,028 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=47) and timeout 30000 to node 2: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:22,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:22,038 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client cfdd2ebd-a50b-40ed-9076-36cf63fe5b42 per-consumer assignment: 02.12.2021 17:25:22,037 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59972) 02.12.2021 17:25:22,051 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client b340ff38-2a93-4102-8a87-c026f0ab26f7 per-consumer assignment: 02.12.2021 17:25:22,051 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:22,051 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462322051 02.12.2021 17:25:22,056 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59949) 02.12.2021 17:25:22,051 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 912326ee-6960-44d6-b637-f6bf4b4fb4d9 per-consumer assignment: 02.12.2021 17:25:22,056 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Received METADATA response from node 2 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=47): org.apache.kafka.common.requests.MetadataResponse@3ae44f91 02.12.2021 17:25:22,057 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:22,057 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Call(callName=metadata, deadlineMs=1638462382000, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:22,057 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462322057 02.12.2021 17:25:22,057 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=209025) 02.12.2021 17:25:22,057 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Removing task 0_8 from staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:22,063 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 due to tasks changing ownership. 02.12.2021 17:25:22,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested client 7f1cad50-1794-4112-bb49-b2fba405b328 to schedule a followup rebalance 02.12.2021 17:25:22,065 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 7f1cad50-1794-4112-bb49-b2fba405b328 per-consumer assignment: 02.12.2021 17:25:22,065 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Removing task 0_5 from staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:22,065 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d due to tasks changing ownership. 02.12.2021 17:25:22,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested client 46d9f289-0630-4b19-b7f2-c59e02299906 to schedule a followup rebalance 02.12.2021 17:25:22,065 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 46d9f289-0630-4b19-b7f2-c59e02299906 per-consumer assignment: 02.12.2021 17:25:22,065 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 8f66a158-1279-467f-a02e-ff9e9f38632b per-consumer assignment: 02.12.2021 17:25:22,066 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 63f94986-6103-497f-970a-6fcbf19c32a5 per-consumer assignment: 02.12.2021 17:25:22,066 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f per-consumer assignment: 02.12.2021 17:25:22,066 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 941317db-575e-41d6-ace9-ee09236fae90 per-consumer assignment: 02.12.2021 17:25:22,066 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:22,066 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Finished assignment for group at generation 57: {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-0], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-2], userDataSize=48), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-7], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-3], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-1], userDataSize=48), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-6], userDataSize=48), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-4], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=Assignment(partitions=[], userDataSize=40)} 02.12.2021 17:25:22,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending leader SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 4, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:22,067 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1267) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]}]} 02.12.2021 17:25:22,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 307 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:22,075 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132465, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:22,075 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132475, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:22,075 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,075 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:22,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:22,076 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59959ms for 60000ms of delay, sleeping for 51ms 02.12.2021 17:25:22,088 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1267): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:22,089 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=663): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,088 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=663): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,089 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,089 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:22,089 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:22,090 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 57 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:22,090 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:22,090 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,090 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:22,090 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:22,090 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,090 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from RUNNING to REBALANCING 02.12.2021 17:25:22,090 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:22,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 285 02.12.2021 17:25:22,089 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:22,089 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,091 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 57 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:22,091 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:22,091 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,091 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,091 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=57, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:22,092 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:22,092 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,092 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:22,092 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 279 02.12.2021 17:25:22,092 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 57 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:22,093 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:22,093 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,093 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,094 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:22,094 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 261 02.12.2021 17:25:22,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,129 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59364ms for 60000ms of delay, sleeping for 646ms 02.12.2021 17:25:22,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:22,191 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 184 02.12.2021 17:25:22,193 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 175 02.12.2021 17:25:22,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 155 02.12.2021 17:25:22,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:22,239 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:22,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,279 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,281 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,281 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 83 02.12.2021 17:25:22,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:22,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 55 02.12.2021 17:25:22,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:22,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:22,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,330 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:22,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:22,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:22,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:22,330 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,332 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,350 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:22,350 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,350 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,350 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,351 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,369 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,369 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,374 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:22,374 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,375 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,375 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:22,375 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:22,375 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,375 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,375 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,376 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from REBALANCING to RUNNING 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:22,376 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:22,376 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 57 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:22,376 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:22,377 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:22,377 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:22,378 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1268) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:22,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 488 02.12.2021 17:25:22,392 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:22,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:22,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:22,447 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1031): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:22,447 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:22,447 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:22,447 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:22,447 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=694): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:22,447 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:22,447 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:22,447 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:22,448 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,448 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=695) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 333 02.12.2021 17:25:22,447 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=670): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:22,447 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,448 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1032) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,448 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:22,450 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:22,450 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:22,450 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,450 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 381 02.12.2021 17:25:22,450 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=671) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 327 02.12.2021 17:25:22,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:22,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:22,485 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1032): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:22,485 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=671): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,485 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,485 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:22,485 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 188 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:22,485 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:22,485 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,485 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,485 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:22,485 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,485 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from RUNNING to REBALANCING 02.12.2021 17:25:22,486 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=695): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,487 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,487 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:22,488 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 188 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:22,488 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:22,488 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,489 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,489 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:22,489 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,485 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:22,485 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:22,489 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:22,489 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:22,489 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:22,489 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=188, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:22,490 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 188 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:22,491 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:22,491 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,491 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:22,491 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:22,491 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,491 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:22,491 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 345 02.12.2021 17:25:22,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:22,533 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:22,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:22,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:22,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 190 02.12.2021 17:25:22,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 191 02.12.2021 17:25:22,592 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 238 02.12.2021 17:25:22,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,595 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,634 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:22,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:22,671 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:22,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 90 02.12.2021 17:25:22,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 90 02.12.2021 17:25:22,692 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 138 02.12.2021 17:25:22,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:22,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:22,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:22,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:22,776 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59943ms for 60000ms of delay, sleeping for 67ms 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,780 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,781 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,781 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:22,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,782 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,792 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 38 02.12.2021 17:25:22,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:22,830 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,830 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,830 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:22,831 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,831 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,831 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:22,831 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:22,831 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,831 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,831 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,831 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from REBALANCING to RUNNING 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:22,832 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 188 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:22,832 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:22,832 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1033) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:22,832 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,837 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:22,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,838 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 428 02.12.2021 17:25:22,844 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58617ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:22,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,851 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,870 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,870 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:22,881 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:22,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:22,893 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:22,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,894 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:22,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:22,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:22,897 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:22,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:22,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 328 02.12.2021 17:25:22,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:22,971 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:22,988 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 293 02.12.2021 17:25:22,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 290 02.12.2021 17:25:23,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:23,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 226 02.12.2021 17:25:23,053 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:23,072 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:23,093 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 188 02.12.2021 17:25:23,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 188 02.12.2021 17:25:23,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:23,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,098 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,098 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:23,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 124 02.12.2021 17:25:23,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:23,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:23,193 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:23,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:23,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:23,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 24 02.12.2021 17:25:23,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:23,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,266 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:23,267 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:23,267 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=656) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:23,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:23,267 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=656): org.apache.kafka.common.requests.HeartbeatResponse@6b1ad3e7 02.12.2021 17:25:23,267 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:23,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 71 02.12.2021 17:25:23,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 70 02.12.2021 17:25:23,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:23,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,281 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,290 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:23,290 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:23,290 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=656) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:23,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 308 02.12.2021 17:25:23,291 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=656): org.apache.kafka.common.requests.HeartbeatResponse@523d5010 02.12.2021 17:25:23,291 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:23,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 307 02.12.2021 17:25:23,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 307 02.12.2021 17:25:23,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,333 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:23,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:23,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:23,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:23,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,354 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,356 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,370 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,371 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,383 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:23,382 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:23,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 206 02.12.2021 17:25:23,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:23,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,405 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:23,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:23,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:23,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:23,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:23,472 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:23,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:23,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:23,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 106 02.12.2021 17:25:23,544 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:23,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:23,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:23,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:23,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:23,592 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 6 02.12.2021 17:25:23,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:23,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,602 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,602 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:23,646 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:23,665 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 191 02.12.2021 17:25:23,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:23,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:23,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:23,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:23,746 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:23,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 89 02.12.2021 17:25:23,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,782 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:23,785 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,784 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,793 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,803 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:23,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,834 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:23,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:23,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:23,834 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,845 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:23,845 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:23,844 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59617ms delay out of 60000ms. Remaining 383ms. Executor queue at 0 02.12.2021 17:25:23,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:23,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,849 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,850 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58775ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:23,845 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=954) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:23,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:23,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,864 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,866 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,881 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:23,888 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,888 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 395 02.12.2021 17:25:23,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:23,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,905 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:23,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:23,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,917 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:23,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:23,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:23,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,950 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=954): org.apache.kafka.common.requests.HeartbeatResponse@4dd86840 02.12.2021 17:25:23,950 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:23,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:23,966 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:24,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 292 02.12.2021 17:25:24,004 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:24,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 295 02.12.2021 17:25:24,019 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 392 02.12.2021 17:25:24,058 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:24,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:24,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 188 02.12.2021 17:25:24,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:24,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,109 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:24,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 291 02.12.2021 17:25:24,162 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:24,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:24,205 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:24,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:24,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:24,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 190 02.12.2021 17:25:24,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:24,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,304 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:24,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:24,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,335 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:24,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:24,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:24,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,350 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:24,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,368 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,369 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:24,410 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:24,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:24,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,412 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:24,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,430 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:24,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:24,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:24,431 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:24,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:24,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 290 02.12.2021 17:25:24,510 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:24,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:24,521 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 391 02.12.2021 17:25:24,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:24,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:24,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 189 02.12.2021 17:25:24,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:24,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,612 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,613 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,615 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 189 02.12.2021 17:25:24,622 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 290 02.12.2021 17:25:24,651 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:24,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:24,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:24,713 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:24,715 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 89 02.12.2021 17:25:24,723 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 190 02.12.2021 17:25:24,752 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:24,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:24,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,795 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,805 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,813 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:24,824 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 89 02.12.2021 17:25:24,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,836 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:24,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:24,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:24,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,850 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59775ms delay out of 60000ms. Remaining 225ms. Executor queue at 0 02.12.2021 17:25:24,851 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59398ms for 60000ms of delay, sleeping for 612ms 02.12.2021 17:25:24,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,851 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,870 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,870 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 86 02.12.2021 17:25:24,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:24,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,913 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,913 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 42 02.12.2021 17:25:24,905 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,914 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:24,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,933 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:24,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:24,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:24,933 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:24,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,956 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:24,958 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 57 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:24,958 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=664) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:24,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,955 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:24,964 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 57 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:24,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 412 02.12.2021 17:25:24,965 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=664): org.apache.kafka.common.requests.HeartbeatResponse@157ee3cd 02.12.2021 17:25:24,965 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=664) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:24,965 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:24,969 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 57 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:24,969 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:24,969 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:24,966 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 458 02.12.2021 17:25:24,970 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=664): org.apache.kafka.common.requests.HeartbeatResponse@1feadb31 02.12.2021 17:25:24,970 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:24,970 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:24,970 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=665) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:24,971 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:24,970 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 57 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:24,971 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:24,971 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:24,971 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:24,971 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=665) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:24,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 441 02.12.2021 17:25:24,996 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:25,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:25,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 291 02.12.2021 17:25:25,053 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:25,058 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:25,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:25,098 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:25,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:25,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,115 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 189 02.12.2021 17:25:25,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:25,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:25,216 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:25,216 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 89 02.12.2021 17:25:25,254 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:25,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,295 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 151 02.12.2021 17:25:25,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,306 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 144 02.12.2021 17:25:25,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:25,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,338 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:25,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,352 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,370 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,371 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 51 02.12.2021 17:25:25,406 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 44 02.12.2021 17:25:25,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:25,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,415 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,415 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:25,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:25,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,436 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:25,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 30189ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:25,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:25,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,448 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,448 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 188 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,448 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=696) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null} 02.12.2021 17:25:25,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:25,449 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=696): org.apache.kafka.common.requests.HeartbeatResponse@5e98a46 02.12.2021 17:25:25,450 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:25,450 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 188 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:25,450 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:25,450 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:25,451 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:25,451 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,452 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=697) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:25,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 343 02.12.2021 17:25:25,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:25,451 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 188 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,453 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=672) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null} 02.12.2021 17:25:25,454 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 354 02.12.2021 17:25:25,454 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=672): org.apache.kafka.common.requests.HeartbeatResponse@2502652b 02.12.2021 17:25:25,454 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:25,455 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 188 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:25,456 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:25,456 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:25,456 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,456 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=673) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:25,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 350 02.12.2021 17:25:25,468 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59207ms for 60000ms of delay, sleeping for 803ms 02.12.2021 17:25:25,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:25,548 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:25,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:25,554 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:25,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:25,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 393 02.12.2021 17:25:25,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:25,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,796 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:25,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,816 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,817 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,832 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:25,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,844 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:25,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,845 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:25,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,856 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 410 02.12.2021 17:25:25,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,872 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,872 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,925 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:25,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,949 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:25,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 310 02.12.2021 17:25:25,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 30706ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:25,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:25,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,959 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:26,058 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 208 02.12.2021 17:25:26,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:26,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,125 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 165 02.12.2021 17:25:26,163 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:26,226 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:26,263 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 3 02.12.2021 17:25:26,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,266 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:26,267 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:26,267 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=657) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:26,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:26,267 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=657): org.apache.kafka.common.requests.HeartbeatResponse@bd7a041 02.12.2021 17:25:26,267 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:26,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:26,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:26,272 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58368ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:26,290 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:26,290 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=657) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:26,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 334 02.12.2021 17:25:26,291 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=657): org.apache.kafka.common.requests.HeartbeatResponse@5ce35427 02.12.2021 17:25:26,291 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:26,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 334 02.12.2021 17:25:26,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 334 02.12.2021 17:25:26,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,297 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:26,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,330 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,345 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:26,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:26,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:26,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:26,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:26,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,356 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,356 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:26,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,373 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 233 02.12.2021 17:25:26,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:26,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,429 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,430 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,457 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 31212ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:26,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:26,460 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,494 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 131 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:26,595 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 31 02.12.2021 17:25:26,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,626 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:26,659 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:26,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:26,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:26,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,805 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,806 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:26,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:26,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,833 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,833 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,849 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:26,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:26,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:26,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:26,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,858 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,859 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:26,860 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:26,865 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:26,865 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=955) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:26,881 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:26,881 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,881 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,887 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,928 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:26,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:26,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,934 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,960 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:26,966 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=955): org.apache.kafka.common.requests.HeartbeatResponse@3389efa3 02.12.2021 17:25:26,966 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:26,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,967 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:26,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:26,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:26,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 31719ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:26,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:26,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:26,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:27,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:27,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,128 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:27,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:27,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:27,272 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59368ms delay out of 60000ms. Remaining 632ms. Executor queue at 0 02.12.2021 17:25:27,274 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58960ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:27,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,307 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:27,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,334 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,351 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:27,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:27,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:27,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:27,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,359 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:27,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,360 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:27,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,395 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,432 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:27,439 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:27,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,442 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:27,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 32233ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:27,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:27,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:27,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:27,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,630 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,630 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,630 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,630 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,630 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,630 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,630 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:27,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:27,764 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:27,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:27,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,826 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:27,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,832 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:27,842 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,848 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,848 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:27,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,860 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:27,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:27,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:27,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:27,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,862 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:27,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,898 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,906 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,906 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:27,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,943 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,945 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:27,968 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:27,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 32747ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:27,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:27,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,050 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:28,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:28,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:28,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,134 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,135 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,168 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:28,235 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:28,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:28,275 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59961ms delay out of 60000ms. Remaining 39ms. Executor queue at 0 02.12.2021 17:25:28,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Committing all active tasks [0_8] and standby tasks [] since 31812ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,275 DEBUG [org.apache.kafka.streams.processor.internals.RecordCollectorImpl:247] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Flushing record collector 02.12.2021 17:25:28,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:381] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Prepared RUNNING task for committing 02.12.2021 17:25:28,275 TRACE [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1157] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending OffsetCommit request with {redacted.Cloud.Message.Callback.Retry1-8=OffsetAndMetadata{offset=1132475, leaderEpoch=null, metadata='AQAAAX179iOa'}} to coordinator kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:28,275 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending OFFSET_COMMIT request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=956) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null,topics=[{name=redacted.Cloud.Message.Callback.Retry1,partitions=[{partition_index=8,committed_offset=1132475,committed_leader_epoch=-1,committed_metadata=AQAAAX179iOa}]}]} 02.12.2021 17:25:28,293 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received OFFSET_COMMIT response from node 2147483644 for request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=956): OffsetCommitResponseData(throttleTimeMs=0, topics=[OffsetCommitResponseTopic(name='redacted.Cloud.Message.Callback.Retry1', partitions=[OffsetCommitResponsePartition(partitionIndex=8, errorCode=0)])]) 02.12.2021 17:25:28,293 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1185] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Committed offset 1132475 for partition redacted.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:28,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:808] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Checkpointable offsets {redacted.Cloud.Message.Callback.Retry1-8=1132474} 02.12.2021 17:25:28,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:465] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Finalized commit for RUNNING task with eos false enforce checkpoint false 02.12.2021 17:25:28,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:684] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Committed all active tasks [0_8] and standby tasks [] in 0ms 02.12.2021 17:25:28,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 8 iterations. 02.12.2021 17:25:28,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 8 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:28,295 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132475, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:28,295 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132485, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:28,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:28,296 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:28,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,296 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:28,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 8 iterations. 02.12.2021 17:25:28,296 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59592ms for 60000ms of delay, sleeping for 418ms 02.12.2021 17:25:28,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:28,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,329 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:28,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:28,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,353 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,361 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:28,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:28,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:28,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,367 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30164ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,368 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:28,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,423 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,450 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,450 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,436 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:28,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:28,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,496 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:28,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:28,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 33248ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:28,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:28,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:28,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,637 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30289ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,637 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:28,717 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59012ms for 60000ms of delay, sleeping for 998ms 02.12.2021 17:25:28,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:28,770 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:28,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:28,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,831 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,832 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:28,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,854 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,862 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:28,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:28,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:28,862 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,869 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 395 02.12.2021 17:25:28,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,937 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,937 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:28,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,951 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,971 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:28,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,999 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:28,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:28,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 33751ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:29,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,044 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:29,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:29,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,138 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,138 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 152 02.12.2021 17:25:29,173 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:29,239 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 51 02.12.2021 17:25:29,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,267 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:29,267 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:29,267 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=658) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:29,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 104 02.12.2021 17:25:29,267 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=658): org.apache.kafka.common.requests.HeartbeatResponse@22338799 02.12.2021 17:25:29,268 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:29,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:29,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:29,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,290 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:29,290 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:29,290 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=658) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:29,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 348 02.12.2021 17:25:29,291 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=658): org.apache.kafka.common.requests.HeartbeatResponse@6fa771b8 02.12.2021 17:25:29,291 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:29,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 347 02.12.2021 17:25:29,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 347 02.12.2021 17:25:29,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,333 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,355 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,363 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:29,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:29,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:29,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,368 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 3 02.12.2021 17:25:29,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,372 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:29,391 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 247 02.12.2021 17:25:29,439 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,439 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,439 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,440 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:29,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,454 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,454 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,473 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:29,477 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=697): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:29,478 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1033): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:29,478 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=673): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:29,479 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:29,479 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:29,479 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:29,479 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:29,479 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=674) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,assignments=[]} 02.12.2021 17:25:29,479 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 376 02.12.2021 17:25:29,478 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:29,480 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=698) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:29,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 353 02.12.2021 17:25:29,479 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:29,484 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:29,484 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:29,486 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:29,486 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1034) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,assignments=[]} 02.12.2021 17:25:29,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 375 02.12.2021 17:25:29,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 142 02.12.2021 17:25:29,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:29,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,502 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:29,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:29,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 34255ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:29,503 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,538 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=698): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,541 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,541 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:29,538 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=674): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,541 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,538 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1034): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:29,541 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:29,541 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:29,541 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=189, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:29,542 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 189 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:29,542 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:29,542 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:29,542 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:29,542 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:29,542 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:29,542 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from RUNNING to REBALANCING 02.12.2021 17:25:29,542 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:29,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 314 02.12.2021 17:25:29,541 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 189 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:29,542 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:29,542 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:29,543 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:29,543 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:29,543 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:29,543 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:29,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 322 02.12.2021 17:25:29,541 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 189 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:29,544 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:29,544 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:29,544 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:29,544 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:29,545 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:29,545 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:29,545 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 292 02.12.2021 17:25:29,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:29,597 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 41 02.12.2021 17:25:29,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:29,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,646 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,645 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 188 02.12.2021 17:25:29,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 220 02.12.2021 17:25:29,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 212 02.12.2021 17:25:29,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,652 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:29,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:29,724 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59816ms for 60000ms of delay, sleeping for 194ms 02.12.2021 17:25:29,748 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 115 02.12.2021 17:25:29,748 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 107 02.12.2021 17:25:29,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 86 02.12.2021 17:25:29,752 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:29,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:29,834 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:29,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:29,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,835 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 7 02.12.2021 17:25:29,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 15 02.12.2021 17:25:29,852 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:29,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:29,859 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:29,864 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:29,867 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:29,867 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from REBALANCING to RUNNING 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,867 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,868 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 189 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:29,868 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:29,868 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1035) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:29,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,871 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:29,871 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:29,871 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=957) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:29,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,883 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:29,924 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59738ms for 60000ms of delay, sleeping for 272ms 02.12.2021 17:25:29,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:29,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:29,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,941 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,948 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:29,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,953 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:29,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,956 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,960 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:29,972 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=957): org.apache.kafka.common.requests.HeartbeatResponse@4f96fa00 02.12.2021 17:25:29,981 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:30,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 376 02.12.2021 17:25:30,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:30,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,007 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:30,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [] and standby tasks [] since 34760ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:30,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,008 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,036 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:30,055 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:30,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 295 02.12.2021 17:25:30,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 276 02.12.2021 17:25:30,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:30,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:30,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,154 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:30,179 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=665): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:30,179 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1268): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0])]) 02.12.2021 17:25:30,179 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:30,179 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:30,179 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:30,179 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:30,179 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0])]) 02.12.2021 17:25:30,180 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:30,180 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:30,180 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=666) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,assignments=[]} 02.12.2021 17:25:30,180 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@75135211, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@182140cf, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2618480d, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@df98b51, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6215d325, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@70a0c090, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@149ca6a9, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@55fe1468, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@19be6909, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6fec9eee, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1e34d454, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3bccdeae, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@51501eff, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5a40f1d5, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@626e08bd, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3f8a93f0, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6e7cc518, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@336f4f5b, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2156826b, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4aa880a2, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@42f4c8ea, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@ba85278, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1e78dcbc, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@105be241, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@748cdb38, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3d7e6c43, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@653bca6c} 02.12.2021 17:25:30,180 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=665): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:30,180 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:30,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,180 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 280 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,180 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,181 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=666) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:30,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 259 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Constructed client metadata {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, b340ff38-2a93-4102-8a87-c026f0ab26f7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 912326ee-6960-44d6-b637-f6bf4b4fb4d9=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7f1cad50-1794-4112-bb49-b2fba405b328=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 46d9f289-0630-4b19-b7f2-c59e02299906=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 8f66a158-1279-467f-a02e-ff9e9f38632b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 63f94986-6103-497f-970a-6fcbf19c32a5=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 941317db-575e-41d6-ace9-ee09236fae90=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Queueing Call(callName=metadata, deadlineMs=1638462390184, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:30,184 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_0]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:30,184 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462390184, tries=0, nextAllowedTryMs=0)] at 1638462330184 02.12.2021 17:25:30,186 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Metadata is ready to use. 02.12.2021 17:25:30,186 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Found least loaded node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) connected with no in-flight requests 02.12.2021 17:25:30,186 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Assigned Call(callName=metadata, deadlineMs=1638462390184, tries=0, nextAllowedTryMs=0) to node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) 02.12.2021 17:25:30,185 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_0]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:30,186 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:30,186 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client cfdd2ebd-a50b-40ed-9076-36cf63fe5b42 per-consumer assignment: 02.12.2021 17:25:30,186 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-3.data.sta.redacted:9093 (id: 2 rack: null). correlationId=48 02.12.2021 17:25:30,187 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=48) and timeout 30000 to node 2: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:30,187 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=60000) 02.12.2021 17:25:30,187 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:30,187 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462330187 02.12.2021 17:25:30,187 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59997) 02.12.2021 17:25:30,186 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client b340ff38-2a93-4102-8a87-c026f0ab26f7 per-consumer assignment: 02.12.2021 17:25:30,187 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Received METADATA response from node 2 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=48): org.apache.kafka.common.requests.MetadataResponse@687d9743 02.12.2021 17:25:30,187 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:30,187 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Call(callName=metadata, deadlineMs=1638462390184, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:30,187 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462330187 02.12.2021 17:25:30,187 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=200895) 02.12.2021 17:25:30,187 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 912326ee-6960-44d6-b637-f6bf4b4fb4d9 per-consumer assignment: 02.12.2021 17:25:30,188 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 7f1cad50-1794-4112-bb49-b2fba405b328 per-consumer assignment: 02.12.2021 17:25:30,188 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Removing task 0_5 from staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:30,188 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d due to tasks changing ownership. 02.12.2021 17:25:30,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested client 46d9f289-0630-4b19-b7f2-c59e02299906 to schedule a followup rebalance 02.12.2021 17:25:30,188 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 46d9f289-0630-4b19-b7f2-c59e02299906 per-consumer assignment: 02.12.2021 17:25:30,189 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 8f66a158-1279-467f-a02e-ff9e9f38632b per-consumer assignment: 02.12.2021 17:25:30,189 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 63f94986-6103-497f-970a-6fcbf19c32a5 per-consumer assignment: 02.12.2021 17:25:30,190 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f per-consumer assignment: 02.12.2021 17:25:30,190 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 941317db-575e-41d6-ace9-ee09236fae90 per-consumer assignment: 02.12.2021 17:25:30,191 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:30,191 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Finished assignment for group at generation 58: {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-0], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-2], userDataSize=48), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-7], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-3], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-1], userDataSize=48), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-6], userDataSize=48), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-4], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-8], userDataSize=48), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=Assignment(partitions=[], userDataSize=40)} 02.12.2021 17:25:30,193 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending leader SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 4, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:30,193 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1269) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]}]} 02.12.2021 17:25:30,194 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 314 02.12.2021 17:25:30,199 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59426ms for 60000ms of delay, sleeping for 584ms 02.12.2021 17:25:30,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 174 02.12.2021 17:25:30,218 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1269): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,218 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,218 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:30,218 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 58 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:30,218 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=666): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,218 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,218 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:30,218 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 58 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:30,218 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:30,218 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:30,218 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:30,219 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:30,219 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:30,219 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from RUNNING to REBALANCING 02.12.2021 17:25:30,219 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:30,219 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 242 02.12.2021 17:25:30,218 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=666): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,218 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:30,219 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-8], userDataSize=48) 02.12.2021 17:25:30,219 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,219 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=58, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:30,219 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:30,219 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 58 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:30,219 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:30,219 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:30,219 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:30,220 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:30,220 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:30,220 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:30,220 DEBUG [org.apache.kafka.streams.processor.internals.ProcessorStateManager:192] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Created state store manager for task 0_8 02.12.2021 17:25:30,220 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 222 02.12.2021 17:25:30,220 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.s.dropped-records 02.12.2021 17:25:30,221 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=dropped-records-rate, group=stream-task-metrics, description=The average number of dropped records per second, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,221 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=dropped-records-total, group=stream-task-metrics, description=The total number of dropped records, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,221 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.s.enforced-processing 02.12.2021 17:25:30,221 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=enforced-processing-rate, group=stream-task-metrics, description=The average number of occurrences of enforced-processing operations per second, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=enforced-processing-total, group=stream-task-metrics, description=The total number of occurrences of enforced-processing operations, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.s.active-process-ratio 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=active-process-ratio, group=stream-task-metrics, description=The fraction of time the thread spent on processing this task among all assigned active tasks, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.s.process-latency 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=process-latency-avg, group=stream-task-metrics, description=The average latency of calls to process, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=process-latency-max, group=stream-task-metrics, description=The maximum latency of calls to process, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.s.punctuate 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=punctuate-latency-avg, group=stream-task-metrics, description=The average latency of calls to punctuate, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=punctuate-latency-max, group=stream-task-metrics, description=The maximum latency of calls to punctuate, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=punctuate-rate, group=stream-task-metrics, description=The average number of calls to punctuate per second, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=punctuate-total, group=stream-task-metrics, description=The total number of calls to punctuate, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.s.active-buffer-count 02.12.2021 17:25:30,222 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=active-buffer-count, group=stream-task-metrics, description=The count of buffered records that are polled from consumer and not yet processed for this active task, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Processor.s.Processor-record-e2e-latency 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=record-e2e-latency-avg, group=stream-processor-node-metrics, description=The average end-to-end latency of a record, measuring by comparing the record timestamp with the system time when it has been fully processed by the node, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8, processor-node-id=Processor}] 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=record-e2e-latency-max, group=stream-processor-node-metrics, description=The maximum end-to-end latency of a record, measuring by comparing the record timestamp with the system time when it has been fully processed by the node, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8, processor-node-id=Processor}] 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=record-e2e-latency-min, group=stream-processor-node-metrics, description=The minimum end-to-end latency of a record, measuring by comparing the record timestamp with the system time when it has been fully processed by the node, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8, processor-node-id=Processor}] 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Source.s.Source-record-e2e-latency 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=record-e2e-latency-avg, group=stream-processor-node-metrics, description=The average end-to-end latency of a record, measuring by comparing the record timestamp with the system time when it has been fully processed by the node, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8, processor-node-id=Source}] 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=record-e2e-latency-max, group=stream-processor-node-metrics, description=The maximum end-to-end latency of a record, measuring by comparing the record timestamp with the system time when it has been fully processed by the node, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8, processor-node-id=Source}] 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=record-e2e-latency-min, group=stream-processor-node-metrics, description=The minimum end-to-end latency of a record, measuring by comparing the record timestamp with the system time when it has been fully processed by the node, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8, processor-node-id=Source}] 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.s.record-lateness 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=record-lateness-avg, group=stream-task-metrics, description=The observed average lateness of records in milliseconds, measured by comparing the record timestamp with the current stream time, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,223 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=record-lateness-max, group=stream-task-metrics, description=The observed maximum lateness of records in milliseconds, measured by comparing the record timestamp with the current stream time, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,224 DEBUG [org.apache.kafka.streams.processor.internals.ProcessorStateManager:208] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Register global stores [] 02.12.2021 17:25:30,224 TRACE [org.apache.kafka.streams.processor.internals.StreamThread:246] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Created task 0_8 with assigned partitions [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:30,224 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: redacted.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:30,224 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:30,224 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:30,224 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Fetching committed offsets for partitions: [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:30,224 TRACE [org.apache.kafka.common.requests.OffsetFetchRequest:93] Fallback the requireStable flag to false as broker only supports OffsetFetchRequest version 5. Need v7 or newer to enable this feature 02.12.2021 17:25:30,224 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending OFFSET_FETCH request with header RequestHeader(apiKey=OFFSET_FETCH, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1270) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,topics=[{name=redacted.Cloud.Message.Callback.Retry3,partition_indexes=[8]}]} 02.12.2021 17:25:30,225 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received OFFSET_FETCH response from node 2147483643 for request with header RequestHeader(apiKey=OFFSET_FETCH, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1270): org.apache.kafka.common.requests.OffsetFetchResponse@334a8b4e 02.12.2021 17:25:30,225 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:820] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Setting offset for partition redacted.Cloud.Message.Callback.Retry3-8 to the committed offset FetchPosition{offset=953637, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null)], epoch=absent}} 02.12.2021 17:25:30,225 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=preferred-read-replica, group=consumer-fetch-manager-metrics, description=The current read replica for the partition, or -1 if reading from leader, tags={client-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, topic=redacted_Cloud_Message_Callback_Retry3, partition=8}] 02.12.2021 17:25:30,225 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 100 02.12.2021 17:25:30,225 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 283 02.12.2021 17:25:30,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:30,254 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:30,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:30,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:30,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 141 02.12.2021 17:25:30,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 121 02.12.2021 17:25:30,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 182 02.12.2021 17:25:30,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,336 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:30,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,360 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,369 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:30,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,369 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,387 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,387 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 20 02.12.2021 17:25:30,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 39 02.12.2021 17:25:30,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 82 02.12.2021 17:25:30,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:30,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,443 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:30,444 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,444 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,444 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:30,444 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:30,444 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,444 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,444 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,457 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:30,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:30,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,465 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:30,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,466 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,487 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:30,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,510 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:30,510 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,510 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,510 INFO [org.apache.kafka.streams.processor.internals.StreamTask:218] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Initialized 02.12.2021 17:25:30,510 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Fetching committed offsets for partitions: [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:30,510 TRACE [org.apache.kafka.common.requests.OffsetFetchRequest:93] Fallback the requireStable flag to false as broker only supports OffsetFetchRequest version 5. Need v7 or newer to enable this feature 02.12.2021 17:25:30,511 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending OFFSET_FETCH request with header RequestHeader(apiKey=OFFSET_FETCH, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1271) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,topics=[{name=redacted.Cloud.Message.Callback.Retry3,partition_indexes=[8]}]} 02.12.2021 17:25:30,511 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received OFFSET_FETCH response from node 2147483643 for request with header RequestHeader(apiKey=OFFSET_FETCH, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1271): org.apache.kafka.common.requests.OffsetFetchResponse@3f05a513 02.12.2021 17:25:30,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:839] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] A committed timestamp was detected: setting the partition time of partition redacted.Cloud.Message.Callback.Retry3-8 to 1638462085832 in stream task 0_8 02.12.2021 17:25:30,511 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:868] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Initializing processor nodes of the topology 02.12.2021 17:25:30,512 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.s.process 02.12.2021 17:25:30,512 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=process-rate, group=stream-task-metrics, description=The average number of calls to process per second, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,512 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=process-total, group=stream-task-metrics, description=The total number of calls to process, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8}] 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Source.s.process 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=process-rate, group=stream-processor-node-metrics, description=The average number of calls to process per second, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8, processor-node-id=Source}] 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=process-total, group=stream-processor-node-metrics, description=The total number of calls to process, tags={thread-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1, task-id=0_8, processor-node-id=Source}] 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Source.s.punctuate 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Source.s.create 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Source.s.destroy 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.ThrottleableProcessor.s.process 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.ThrottleableProcessor.s.punctuate 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.ThrottleableProcessor.s.create 02.12.2021 17:25:30,513 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.ThrottleableProcessor.s.destroy 02.12.2021 17:25:30,513 INFO [com.sigfox.kafka.throttleable.ThrottleableStreamProcessor:36] Stream 'staging-message-processor-retry3' task '0_8': pausable processor starting 02.12.2021 17:25:30,514 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Processor.s.process 02.12.2021 17:25:30,514 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Processor.s.punctuate 02.12.2021 17:25:30,514 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Processor.s.create 02.12.2021 17:25:30,514 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name internal.staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1.task.0_8.node.Processor.s.destroy 02.12.2021 17:25:30,514 INFO [org.apache.kafka.streams.processor.internals.StreamTask:239] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Restored and ready to run 02.12.2021 17:25:30,514 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,515 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,515 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from REBALANCING to RUNNING 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Committing all active tasks [0_8] and standby tasks [] since 35267ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:1199] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Added READ_UNCOMMITTED fetch request for partition redacted.Cloud.Message.Callback.Retry3-8 at position FetchPosition{offset=953637, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null)], epoch=absent}} to node kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.clients.FetchSessionHandler:259] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Built incremental fetch (sessionId=1808440997, epoch=573) for node 3. Added (), altered (redacted.Cloud.Message.Callback.Retry3-8), removed () out of (OSS.STAGING.Cloud.Message.Callback.Retry3-8) 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:266] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(redacted.Cloud.Message.Callback.Retry3-8), toForget=(), implied=()) to broker kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) 02.12.2021 17:25:30,515 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,515 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending FETCH request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1272) and timeout 30000 to node 3: {replica_id=-1,max_wait_ms=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=1808440997,session_epoch=573,topics=[{topic=redacted.Cloud.Message.Callback.Retry3,partitions=[{partition=8,current_leader_epoch=-1,fetch_offset=953637,log_start_offset=-1,partition_max_bytes=1048576}]}],forgotten_topics_data=[],rack_id=} 02.12.2021 17:25:30,516 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:30,516 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:30,516 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 498 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 498 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 498 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 498 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 498 02.12.2021 17:25:30,518 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry3-8 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:30,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 497 02.12.2021 17:25:30,518 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received FETCH response from node 3 for request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1272): org.apache.kafka.common.requests.FetchResponse@37fa14fc 02.12.2021 17:25:30,518 DEBUG [org.apache.kafka.clients.FetchSessionHandler:463] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Node 3 sent an incremental fetch response with throttleTimeMs = 0 for session 1808440997 with response=(redacted.Cloud.Message.Callback.Retry3-8) 02.12.2021 17:25:30,518 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:315] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Fetch READ_UNCOMMITTED at offset 953637 for partition redacted.Cloud.Message.Callback.Retry3-8 returned fetch data (error=NONE, highWaterMark=954291, lastStableOffset = 954291, logStartOffset = 0, preferredReadReplica = absent, abortedTransactions = null, divergingEpoch =Optional.empty, recordsSizeInBytes=288902) 02.12.2021 17:25:30,518 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1251] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Preparing to read 288902 bytes of data for partition redacted.Cloud.Message.Callback.Retry3-8 with offset FetchPosition{offset=953637, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null)], epoch=absent}} 02.12.2021 17:25:30,518 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1275] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating high watermark for partition redacted.Cloud.Message.Callback.Retry3-8 to 954291 02.12.2021 17:25:30,520 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1280] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating log start offset for partition redacted.Cloud.Message.Callback.Retry3-8 to 0 02.12.2021 17:25:30,520 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1285] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating last stable offset for partition redacted.Cloud.Message.Callback.Retry3-8 to 954291 02.12.2021 17:25:30,520 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953637, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:30,520 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953647, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:30,521 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name redacted.Cloud.Message.Callback.Retry3-8.records-lag 02.12.2021 17:25:30,522 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=records-lag, group=consumer-fetch-manager-metrics, description=The latest lag of the partition, tags={client-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, topic=redacted_Cloud_Message_Callback_Retry3, partition=8}] 02.12.2021 17:25:30,522 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=records-lag-max, group=consumer-fetch-manager-metrics, description=The max lag of the partition, tags={client-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, topic=redacted_Cloud_Message_Callback_Retry3, partition=8}] 02.12.2021 17:25:30,522 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=records-lag-avg, group=consumer-fetch-manager-metrics, description=The average lag of the partition, tags={client-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, topic=redacted_Cloud_Message_Callback_Retry3, partition=8}] 02.12.2021 17:25:30,522 TRACE [org.apache.kafka.common.metrics.Metrics:430] Added sensor with name redacted.Cloud.Message.Callback.Retry3-8.records-lead 02.12.2021 17:25:30,522 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=records-lead, group=consumer-fetch-manager-metrics, description=The latest lead of the partition, tags={client-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, topic=redacted_Cloud_Message_Callback_Retry3, partition=8}] 02.12.2021 17:25:30,524 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=records-lead-min, group=consumer-fetch-manager-metrics, description=The min lead of the partition, tags={client-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, topic=redacted_Cloud_Message_Callback_Retry3, partition=8}] 02.12.2021 17:25:30,524 TRACE [org.apache.kafka.common.metrics.Metrics:587] Registered metric named MetricName [name=records-lead-avg, group=consumer-fetch-manager-metrics, description=The average lead of the partition, tags={client-id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, topic=redacted_Cloud_Message_Callback_Retry3, partition=8}] 02.12.2021 17:25:30,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 9 ms and fetched 10 records 02.12.2021 17:25:30,526 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:30,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,526 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 20 iterations. 02.12.2021 17:25:30,537 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:30,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 20 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 21 iterations. 02.12.2021 17:25:30,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 21 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,538 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953647, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:30,538 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953657, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:30,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:30,539 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:30,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,539 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 21 iterations. 02.12.2021 17:25:30,546 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239639ms for 240000ms of delay, sleeping for 371ms 02.12.2021 17:25:30,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 393 02.12.2021 17:25:30,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:30,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:30,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:30,588 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:30,637 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:30,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,655 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 293 02.12.2021 17:25:30,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:30,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:30,688 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:30,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:30,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:30,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:30,763 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:30,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:30,785 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59480ms for 60000ms of delay, sleeping for 530ms 02.12.2021 17:25:30,791 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:30,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,836 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:30,857 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:30,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:30,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,864 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:30,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,870 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:30,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:30,887 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,892 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,892 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,892 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,892 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,892 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,892 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,892 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,919 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239945ms for 240000ms of delay, sleeping for 65ms 02.12.2021 17:25:30,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:30,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,949 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,949 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:30,957 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:30,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,974 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,974 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 21 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 22 iterations. 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 22 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,989 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953657, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:30,989 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953667, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:30,989 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 22 iterations. 02.12.2021 17:25:30,990 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239192ms for 240000ms of delay, sleeping for 818ms 02.12.2021 17:25:30,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:31,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 295 02.12.2021 17:25:31,050 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:31,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:31,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:31,075 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:31,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:31,143 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:31,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:31,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:31,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,158 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,160 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:31,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:31,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:31,243 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:31,251 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:31,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:31,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:31,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:31,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:31,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 8 records with 8 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:31,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 9 iterations. 02.12.2021 17:25:31,316 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59227ms for 60000ms of delay, sleeping for 783ms 02.12.2021 17:25:31,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,337 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:31,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:31,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,371 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:31,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:31,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:31,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:31,371 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:31,393 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:31,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,451 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:31,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:31,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,475 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:31,544 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 293 02.12.2021 17:25:31,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:31,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:31,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:31,576 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:31,601 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:31,644 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:31,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:31,660 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,660 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,660 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,660 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:31,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:31,701 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:31,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:31,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:31,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:31,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:31,785 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 190 02.12.2021 17:25:31,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:31,809 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239866ms for 240000ms of delay, sleeping for 144ms 02.12.2021 17:25:31,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,838 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:31,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:31,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,865 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,865 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,873 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:31,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:31,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:31,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:31,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,888 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 87 02.12.2021 17:25:31,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,911 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 356 02.12.2021 17:25:31,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:31,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,951 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:31,966 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:31,968 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239764ms for 240000ms of delay, sleeping for 246ms 02.12.2021 17:25:31,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:31,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,977 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 256 02.12.2021 17:25:32,045 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 293 02.12.2021 17:25:32,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:32,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:32,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:32,078 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:32,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 2 records with 9 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:32,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 10 iterations. 02.12.2021 17:25:32,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 10 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:32,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:32,112 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132485, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:32,112 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132495, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:32,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 8 ms and fetched 10 records 02.12.2021 17:25:32,117 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:32,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,117 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:32,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 10 iterations. 02.12.2021 17:25:32,117 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59085ms for 60000ms of delay, sleeping for 925ms 02.12.2021 17:25:32,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 149 02.12.2021 17:25:32,145 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:32,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:32,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,171 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 119 02.12.2021 17:25:32,178 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:32,219 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239090ms for 240000ms of delay, sleeping for 920ms 02.12.2021 17:25:32,219 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 48 02.12.2021 17:25:32,247 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:32,256 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:32,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:32,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,269 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,269 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:32,269 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=659) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:32,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 142 02.12.2021 17:25:32,271 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=659): org.apache.kafka.common.requests.HeartbeatResponse@6df940b7 02.12.2021 17:25:32,271 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:32,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 140 02.12.2021 17:25:32,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 140 02.12.2021 17:25:32,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 18 02.12.2021 17:25:32,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:32,291 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,291 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:32,291 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=659) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:32,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 380 02.12.2021 17:25:32,291 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=659): org.apache.kafka.common.requests.HeartbeatResponse@6f1c2773 02.12.2021 17:25:32,291 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:32,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 380 02.12.2021 17:25:32,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 380 02.12.2021 17:25:32,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,338 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,338 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 142 02.12.2021 17:25:32,357 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:32,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,366 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 113 02.12.2021 17:25:32,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 39 02.12.2021 17:25:32,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:32,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,376 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:32,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,377 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:32,377 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,379 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:32,379 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:32,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 279 02.12.2021 17:25:32,379 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:32,393 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,412 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,412 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:32,439 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 41 02.12.2021 17:25:32,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,455 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,455 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,467 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 13 02.12.2021 17:25:32,480 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 189 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,480 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=675) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null} 02.12.2021 17:25:32,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:32,481 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=675): org.apache.kafka.common.requests.HeartbeatResponse@4ceacc7b 02.12.2021 17:25:32,484 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:32,484 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 189 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:32,484 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:32,485 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:32,485 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,485 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=676) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:32,485 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 381 02.12.2021 17:25:32,481 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,485 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 189 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,485 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=699) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null} 02.12.2021 17:25:32,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:32,486 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=699): org.apache.kafka.common.requests.HeartbeatResponse@400ff4ca 02.12.2021 17:25:32,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:32,491 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:32,493 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 189 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:32,495 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:32,495 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:32,496 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,496 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=700) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:32,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 341 02.12.2021 17:25:32,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:32,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 396 02.12.2021 17:25:32,580 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:32,581 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:32,585 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:32,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:32,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:32,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 293 02.12.2021 17:25:32,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:32,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:32,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:32,763 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:32,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:32,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:32,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:32,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,839 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,840 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,840 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:32,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,867 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,876 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:32,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:32,885 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,893 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:32,893 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=958) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:32,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 523 ms and fetched 0 records 02.12.2021 17:25:32,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,902 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:32,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:32,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:32,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:32,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:32,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,921 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:32,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,956 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 225 02.12.2021 17:25:32,976 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:32,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,984 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,986 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:32,994 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=958): org.apache.kafka.common.requests.HeartbeatResponse@5a49d42f 02.12.2021 17:25:32,994 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:33,022 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:33,056 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 125 02.12.2021 17:25:33,066 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59934ms for 60000ms of delay, sleeping for 76ms 02.12.2021 17:25:33,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:33,087 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:33,122 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:33,141 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239215ms for 240000ms of delay, sleeping for 795ms 02.12.2021 17:25:33,143 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59368ms for 60000ms of delay, sleeping for 642ms 02.12.2021 17:25:33,159 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 22 02.12.2021 17:25:33,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,174 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,179 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:33,179 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 58 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,179 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,179 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=667) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:33,180 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 306 02.12.2021 17:25:33,180 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=667): org.apache.kafka.common.requests.HeartbeatResponse@4d652cf 02.12.2021 17:25:33,180 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:33,180 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 58 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:33,180 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:33,181 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:33,181 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,181 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=668) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 305 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:33,184 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 58 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,184 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=667) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:33,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 272 02.12.2021 17:25:33,185 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=667): org.apache.kafka.common.requests.HeartbeatResponse@79fca790 02.12.2021 17:25:33,185 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:33,185 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 58 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:33,185 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:33,185 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:33,185 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,185 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=668) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:33,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 271 02.12.2021 17:25:33,223 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:33,230 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:33,230 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 58 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,230 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1273) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null} 02.12.2021 17:25:33,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:33,280 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:33,285 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:33,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:33,330 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1273): org.apache.kafka.common.requests.HeartbeatResponse@44a52e91 02.12.2021 17:25:33,330 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:33,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,342 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,368 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,369 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:33,375 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:33,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,407 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,408 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:33,411 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,426 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:33,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,468 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,476 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:33,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:33,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,489 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,489 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:33,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:33,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:33,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:33,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,687 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,688 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:33,788 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59962ms for 60000ms of delay, sleeping for 48ms 02.12.2021 17:25:33,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:33,829 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:33,843 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59870ms for 60000ms of delay, sleeping for 140ms 02.12.2021 17:25:33,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:33,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,844 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,847 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:33,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,871 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,891 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:33,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:33,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,916 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:33,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:33,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:33,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:33,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,930 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,938 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238748ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:33,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,972 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,974 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,991 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,991 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,991 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59600ms for 60000ms of delay, sleeping for 410ms 02.12.2021 17:25:33,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:34,036 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 394 02.12.2021 17:25:34,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:34,137 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:34,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:34,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,188 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,189 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,237 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:34,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:34,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:34,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:34,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,348 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,372 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,389 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:34,407 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59379ms for 60000ms of delay, sleeping for 631ms 02.12.2021 17:25:34,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:34,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,422 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:34,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:34,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:34,422 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:34,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,431 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,431 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,479 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,479 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,479 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,479 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,479 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,479 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,479 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,489 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:34,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,492 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,532 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:34,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:34,632 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:34,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:34,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,691 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,732 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:34,791 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:34,834 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:34,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,849 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,873 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,873 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:34,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:34,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,923 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:34,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:34,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:34,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,932 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,933 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 336 02.12.2021 17:25:34,938 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239748ms delay out of 240000ms. Remaining 252ms. Executor queue at 0 02.12.2021 17:25:34,942 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239062ms for 240000ms of delay, sleeping for 948ms 02.12.2021 17:25:34,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:34,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,980 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:34,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,993 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 236 02.12.2021 17:25:35,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 10 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:35,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:35,039 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132495, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:35,039 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132505, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:35,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:35,040 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:35,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,040 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:35,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:35,043 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59485ms for 60000ms of delay, sleeping for 525ms 02.12.2021 17:25:35,092 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:35,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 136 02.12.2021 17:25:35,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,192 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,192 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:35,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 35 02.12.2021 17:25:35,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,269 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:35,269 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:35,269 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=660) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:35,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 164 02.12.2021 17:25:35,269 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=660): org.apache.kafka.common.requests.HeartbeatResponse@50412304 02.12.2021 17:25:35,269 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:35,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 164 02.12.2021 17:25:35,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 163 02.12.2021 17:25:35,292 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:35,292 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:35,292 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=660) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:35,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:35,292 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=660): org.apache.kafka.common.requests.HeartbeatResponse@57195750 02.12.2021 17:25:35,292 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:35,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:35,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:35,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,350 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 63 02.12.2021 17:25:35,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,374 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,393 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:35,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:35,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,425 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:35,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:35,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Committing all active tasks [] and standby tasks [] since 30205ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:35,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:35,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:35,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:35,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,434 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:35,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,482 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,494 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,494 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:35,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:35,569 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59416ms for 60000ms of delay, sleeping for 594ms 02.12.2021 17:25:35,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:35,638 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:35,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,693 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:35,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:35,794 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:35,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:35,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,866 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,866 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,875 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,878 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,879 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=700): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:35,880 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:35,880 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:35,880 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:35,880 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:35,880 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=701) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:35,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 486 02.12.2021 17:25:35,891 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239707ms for 240000ms of delay, sleeping for 303ms 02.12.2021 17:25:35,884 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=676): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:35,879 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1035): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:35,894 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:35,897 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:35,897 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:35,891 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:35,898 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:35,898 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:35,898 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:35,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 480 02.12.2021 17:25:35,898 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=677) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,assignments=[]} 02.12.2021 17:25:35,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 480 02.12.2021 17:25:35,903 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:35,903 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:35,904 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=959) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:35,897 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:35,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:35,905 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1036) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,assignments=[]} 02.12.2021 17:25:35,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 19 02.12.2021 17:25:35,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,936 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:35,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:35,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,949 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:35,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:35,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Committing all active tasks [] and standby tasks [] since 30737ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:35,972 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:35,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:35,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1036): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,988 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,982 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=701): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:35,989 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:35,982 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=677): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:35,989 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:35,989 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:35,989 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 190 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:35,989 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:35,989 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:35,989 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:35,989 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:35,989 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:35,989 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from RUNNING to REBALANCING 02.12.2021 17:25:35,989 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:35,990 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 377 02.12.2021 17:25:35,991 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:35,991 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 190 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:35,991 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:35,991 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:35,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:35,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,995 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,991 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:35,997 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:35,997 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:35,997 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:35,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 389 02.12.2021 17:25:35,988 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=190, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:35,998 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 190 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:36,004 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=959): org.apache.kafka.common.requests.HeartbeatResponse@54dd2d4e 02.12.2021 17:25:36,004 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:36,000 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:36,006 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:36,006 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:36,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:36,007 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:36,012 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:36,012 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:36,013 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 492 02.12.2021 17:25:36,048 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 389 02.12.2021 17:25:36,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 276 02.12.2021 17:25:36,098 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 281 02.12.2021 17:25:36,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:36,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 362 02.12.2021 17:25:36,148 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 288 02.12.2021 17:25:36,165 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58980ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:36,193 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 173 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 22 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 23 iterations. 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 23 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 58 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:36,195 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1274) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:36,196 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953667, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:36,196 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953677, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:36,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:36,196 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:36,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,196 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:36,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 23 iterations. 02.12.2021 17:25:36,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:36,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,197 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,197 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239915ms for 240000ms of delay, sleeping for 95ms 02.12.2021 17:25:36,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,198 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 179 02.12.2021 17:25:36,213 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 262 02.12.2021 17:25:36,241 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:36,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:36,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 72 02.12.2021 17:25:36,295 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239107ms for 240000ms of delay, sleeping for 903ms 02.12.2021 17:25:36,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:36,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 79 02.12.2021 17:25:36,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 162 02.12.2021 17:25:36,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:36,366 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:36,366 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:36,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:36,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,367 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,368 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:36,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:36,380 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:36,380 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:36,380 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:36,380 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:36,380 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:36,381 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,381 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:36,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 61 02.12.2021 17:25:36,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:36,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,437 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:36,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:36,479 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:36,479 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:36,479 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:36,480 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:36,480 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from REBALANCING to RUNNING 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Committing all active tasks [] and standby tasks [] since 31260ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:36,480 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:36,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:36,481 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 190 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:36,481 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:36,481 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:36,481 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:36,481 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1037) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:36,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:36,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:36,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,490 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,491 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:36,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,498 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,500 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:36,514 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:36,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:36,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:36,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:36,600 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:36,638 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:36,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:36,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:36,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,699 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,699 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:36,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:36,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:36,799 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:36,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:36,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:36,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,869 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,870 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,885 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,900 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:36,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,946 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:36,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:36,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,983 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:36,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:36,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:36,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,985 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:36,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,992 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:37,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,000 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:37,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:37,076 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:37,088 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:37,101 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:37,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:37,165 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59980ms delay out of 60000ms. Remaining 20ms. Executor queue at 0 02.12.2021 17:25:37,167 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59453ms for 60000ms of delay, sleeping for 557ms 02.12.2021 17:25:37,177 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:37,193 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 192 02.12.2021 17:25:37,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,201 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,201 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,202 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239979ms for 240000ms of delay, sleeping for 31ms 02.12.2021 17:25:37,235 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238838ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:37,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:37,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:37,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:37,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:37,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:37,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,371 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,371 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:37,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,392 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:37,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,446 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,447 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:37,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:37,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,486 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:37,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:37,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:37,487 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:37,487 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 394 02.12.2021 17:25:37,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:37,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,500 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,500 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:37,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,503 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,503 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:37,541 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=668): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:37,541 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:37,541 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:37,541 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:37,541 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=668): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:37,542 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:37,547 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:37,542 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:37,547 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=669) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:37,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 452 02.12.2021 17:25:37,548 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:37,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:37,551 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:37,556 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=669) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,assignments=[]} 02.12.2021 17:25:37,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 447 02.12.2021 17:25:37,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:37,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 293 02.12.2021 17:25:37,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:37,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:37,672 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:37,700 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 192 02.12.2021 17:25:37,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:37,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,704 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,729 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59614ms for 60000ms of delay, sleeping for 396ms 02.12.2021 17:25:37,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:37,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:37,801 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:37,808 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:37,858 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:37,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,872 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,873 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:37,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,894 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,910 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:37,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,946 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 323 02.12.2021 17:25:37,974 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:37,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,989 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:37,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:37,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:37,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:37,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:38,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:38,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,002 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,002 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:38,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,008 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:38,046 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 223 02.12.2021 17:25:38,075 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:38,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:38,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:38,129 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59926ms for 60000ms of delay, sleeping for 84ms 02.12.2021 17:25:38,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 120 02.12.2021 17:25:38,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:38,198 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:38,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:38,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,208 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,208 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:38,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:38,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:38,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:38,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:38,215 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132505, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:38,215 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132515, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:38,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:38,215 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:38,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,216 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:38,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:38,216 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59984ms for 60000ms of delay, sleeping for 26ms 02.12.2021 17:25:38,235 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239838ms delay out of 240000ms. Remaining 162ms. Executor queue at 0 02.12.2021 17:25:38,235 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239592ms for 240000ms of delay, sleeping for 418ms 02.12.2021 17:25:38,243 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59256ms for 60000ms of delay, sleeping for 754ms 02.12.2021 17:25:38,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 19 02.12.2021 17:25:38,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,269 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,269 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:38,270 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=661) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:38,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 177 02.12.2021 17:25:38,270 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=661): org.apache.kafka.common.requests.HeartbeatResponse@66a254e5 02.12.2021 17:25:38,270 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:38,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 176 02.12.2021 17:25:38,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 176 02.12.2021 17:25:38,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:38,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,296 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,296 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:38,296 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=661) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:38,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 411 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=661): org.apache.kafka.common.requests.HeartbeatResponse@111d73f8 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:38,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 411 02.12.2021 17:25:38,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 410 02.12.2021 17:25:38,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:38,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 76 02.12.2021 17:25:38,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,373 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:38,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,396 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 310 02.12.2021 17:25:38,447 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,449 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,449 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:38,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:38,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:38,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,494 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:38,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:38,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:38,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:38,494 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:38,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 206 02.12.2021 17:25:38,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:38,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,504 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:38,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,510 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,511 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:38,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:38,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:38,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 104 02.12.2021 17:25:38,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:38,655 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238798ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:38,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:38,697 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:38,706 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 2 02.12.2021 17:25:38,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,709 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:38,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:38,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:38,797 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:38,810 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:38,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:38,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:38,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,876 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 3 02.12.2021 17:25:38,880 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,894 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 190 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:38,894 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=702) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null} 02.12.2021 17:25:38,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:38,898 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 190 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=678) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null} 02.12.2021 17:25:38,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,894 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=702): org.apache.kafka.common.requests.HeartbeatResponse@6397bf3 02.12.2021 17:25:38,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=678): org.apache.kafka.common.requests.HeartbeatResponse@533b3e5b 02.12.2021 17:25:38,914 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:38,914 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 190 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:38,914 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:38,914 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:38,910 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,915 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:38,910 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:38,915 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=960) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:38,914 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:38,916 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=679) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:38,916 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 482 02.12.2021 17:25:38,915 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 190 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:38,918 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:38,918 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:38,918 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:38,918 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=703) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:38,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 459 02.12.2021 17:25:38,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,950 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,950 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:38,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,997 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:38,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:38,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:38,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:38,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,994 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:39,004 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,005 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,998 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:39,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,011 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:39,014 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,015 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=960): org.apache.kafka.common.requests.HeartbeatResponse@28d52481 02.12.2021 17:25:39,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,021 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:39,027 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58909ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:39,051 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:39,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:39,151 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:39,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:39,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:39,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:39,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,378 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,385 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,387 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,401 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,401 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,411 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:39,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:39,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,500 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:39,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:39,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:39,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:39,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,508 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:39,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,523 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:39,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:39,623 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:39,655 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239798ms delay out of 240000ms. Remaining 202ms. Executor queue at 0 02.12.2021 17:25:39,656 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239694ms for 240000ms of delay, sleeping for 316ms 02.12.2021 17:25:39,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:39,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:39,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,711 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,757 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:39,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:39,857 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:39,892 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:39,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,900 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,903 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,905 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:39,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,960 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,975 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239673ms for 240000ms of delay, sleeping for 337ms 02.12.2021 17:25:40,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,001 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:40,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:40,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:40,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:40,002 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:40,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,009 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:40,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:40,027 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59909ms delay out of 60000ms. Remaining 91ms. Executor queue at 0 02.12.2021 17:25:40,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,027 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:40,030 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59220ms for 60000ms of delay, sleeping for 790ms 02.12.2021 17:25:40,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:40,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:40,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:40,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,212 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,212 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,212 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,212 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,212 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,212 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,212 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:40,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:40,315 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239078ms for 240000ms of delay, sleeping for 932ms 02.12.2021 17:25:40,367 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:40,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:40,407 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,408 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,410 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,411 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,413 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:40,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:40,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,413 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:40,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,468 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,501 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:40,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:40,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:40,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:40,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,511 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,511 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 30 02.12.2021 17:25:40,513 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:40,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,528 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 17 02.12.2021 17:25:40,542 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:40,543 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:40,543 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=670) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:40,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:40,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 468 02.12.2021 17:25:40,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:40,547 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:40,548 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:40,548 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=670) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:40,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 481 02.12.2021 17:25:40,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:40,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:40,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:40,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,713 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,714 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,714 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:40,814 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:40,828 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59760ms for 60000ms of delay, sleeping for 250ms 02.12.2021 17:25:40,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:40,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,915 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,915 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,915 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:40,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,918 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,968 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,977 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:41,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:41,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:41,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,017 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:41,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:41,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:41,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:41,020 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,020 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,019 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:41,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,035 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,035 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:41,079 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 4 fetched records at offset FetchPosition{offset=1132515, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:41,079 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132519, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:1199] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Added READ_UNCOMMITTED fetch request for partition redacted.Cloud.Message.Callback.Retry1-8 at position FetchPosition{offset=1132519, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null)], epoch=absent}} to node kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.clients.FetchSessionHandler:259] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Built incremental fetch (sessionId=1223465210, epoch=238) for node 3. Added (), altered (redacted.Cloud.Message.Callback.Retry1-8), removed () out of (OSS.STAGING.Cloud.Message.Callback.Retry1-8) 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:266] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(redacted.Cloud.Message.Callback.Retry1-8), toForget=(), implied=()) to broker kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) 02.12.2021 17:25:41,079 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending FETCH request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=961) and timeout 30000 to node 3: {replica_id=-1,max_wait_ms=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=1223465210,session_epoch=238,topics=[{topic=redacted.Cloud.Message.Callback.Retry1,partitions=[{partition=8,current_leader_epoch=-1,fetch_offset=1132519,log_start_offset=-1,partition_max_bytes=1048576}]}],forgotten_topics_data=[],rack_id=} 02.12.2021 17:25:41,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 4 records 02.12.2021 17:25:41,080 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 4 02.12.2021 17:25:41,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,080 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:41,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:41,078 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:41,091 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59123ms for 60000ms of delay, sleeping for 887ms 02.12.2021 17:25:41,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:41,124 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received FETCH response from node 3 for request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=961): org.apache.kafka.common.requests.FetchResponse@506b8d61 02.12.2021 17:25:41,125 DEBUG [org.apache.kafka.clients.FetchSessionHandler:463] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Node 3 sent an incremental fetch response with throttleTimeMs = 0 for session 1223465210 with response=(redacted.Cloud.Message.Callback.Retry1-8) 02.12.2021 17:25:41,125 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:315] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Fetch READ_UNCOMMITTED at offset 1132519 for partition redacted.Cloud.Message.Callback.Retry1-8 returned fetch data (error=NONE, highWaterMark=1132683, lastStableOffset = 1132683, logStartOffset = 0, preferredReadReplica = absent, abortedTransactions = null, divergingEpoch =Optional.empty, recordsSizeInBytes=74677) 02.12.2021 17:25:41,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:41,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:41,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,214 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,214 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:41,248 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239748ms for 240000ms of delay, sleeping for 262ms 02.12.2021 17:25:41,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,269 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=662) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=703): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:41,270 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=679): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:41,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 208 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=662): org.apache.kafka.common.requests.HeartbeatResponse@581318e7 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:41,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 208 02.12.2021 17:25:41,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 207 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=704) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:41,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 147 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1037): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:41,272 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:41,272 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:41,275 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:41,272 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:41,275 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:41,275 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:41,275 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1038) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,assignments=[]} 02.12.2021 17:25:41,275 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:41,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 243 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=680) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,assignments=[]} 02.12.2021 17:25:41,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 137 02.12.2021 17:25:41,297 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,297 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:41,297 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=662) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:41,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 418 02.12.2021 17:25:41,297 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=662): org.apache.kafka.common.requests.HeartbeatResponse@45e3ead2 02.12.2021 17:25:41,297 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:41,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 417 02.12.2021 17:25:41,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 417 02.12.2021 17:25:41,299 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1038): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:41,299 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=680): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,299 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=704): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,299 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:41,299 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:41,299 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,299 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:41,299 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,299 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=191, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:41,299 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 191 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:41,299 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 191 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:41,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:41,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,300 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 191 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:41,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:41,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,300 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:41,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:41,300 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:41,300 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,301 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from RUNNING to REBALANCING 02.12.2021 17:25:41,301 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:41,300 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,301 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:41,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 119 02.12.2021 17:25:41,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 116 02.12.2021 17:25:41,301 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:41,303 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:41,303 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,303 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:41,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 220 02.12.2021 17:25:41,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 107 02.12.2021 17:25:41,403 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 116 02.12.2021 17:25:41,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 316 02.12.2021 17:25:41,403 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 12 02.12.2021 17:25:41,401 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 17 02.12.2021 17:25:41,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,419 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:41,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,424 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:41,424 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,424 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,424 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:41,424 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,425 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,425 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,472 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 6 02.12.2021 17:25:41,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:41,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,482 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:41,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 15 02.12.2021 17:25:41,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 210 02.12.2021 17:25:41,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 23 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 24 iterations. 02.12.2021 17:25:41,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 24 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:41,511 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1274): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0])]) 02.12.2021 17:25:41,512 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 31, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0])]) 02.12.2021 17:25:41,512 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:41,513 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:41,513 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5c1c4c49, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@24a28108, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5eac5217, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3b58dd3d, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@598a01b, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3582db61, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@9020b2c, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@744e9e04, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@349670ee, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@17ecfcce, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@e2ece74, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@e90df0b, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@906df37, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@797c5855, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1d96460, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@9b99bd7, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@367912b0, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2d5c4ada, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@67ecd175, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@22ce997c, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5c5ce307, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@139978cc, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@12ff74b3, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@44ac1ae9, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7f539da0, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2754d59e, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4e67672b} 02.12.2021 17:25:41,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:41,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,525 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,525 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from REBALANCING to RUNNING 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:41,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,526 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,527 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:41,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:41,527 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 191 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:41,527 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:41,527 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:41,527 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:41,528 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1039) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:41,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:41,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Constructed client metadata {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, b340ff38-2a93-4102-8a87-c026f0ab26f7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 912326ee-6960-44d6-b637-f6bf4b4fb4d9=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7f1cad50-1794-4112-bb49-b2fba405b328=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 46d9f289-0630-4b19-b7f2-c59e02299906=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 8f66a158-1279-467f-a02e-ff9e9f38632b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 63f94986-6103-497f-970a-6fcbf19c32a5=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 941317db-575e-41d6-ace9-ee09236fae90=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Queueing Call(callName=metadata, deadlineMs=1638462401529, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:41,529 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:41,530 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462401529, tries=0, nextAllowedTryMs=0)] at 1638462341530 02.12.2021 17:25:41,530 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Metadata is ready to use. 02.12.2021 17:25:41,530 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Found least loaded node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) connected with no in-flight requests 02.12.2021 17:25:41,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_0]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:41,530 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Assigned Call(callName=metadata, deadlineMs=1638462401529, tries=0, nextAllowedTryMs=0) to node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) 02.12.2021 17:25:41,531 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_0]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:41,531 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-3.data.sta.redacted:9093 (id: 2 rack: null). correlationId=49 02.12.2021 17:25:41,532 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=49) and timeout 30000 to node 2: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:41,532 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59999) 02.12.2021 17:25:41,532 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:41,533 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462341532 02.12.2021 17:25:41,533 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59997) 02.12.2021 17:25:41,533 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Received METADATA response from node 2 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=49): org.apache.kafka.common.requests.MetadataResponse@33d529e8 02.12.2021 17:25:41,533 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:41,533 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Call(callName=metadata, deadlineMs=1638462401529, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:41,533 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462341533 02.12.2021 17:25:41,533 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=189549) 02.12.2021 17:25:41,532 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:41,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:41,536 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client cfdd2ebd-a50b-40ed-9076-36cf63fe5b42 per-consumer assignment: 02.12.2021 17:25:41,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,538 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client b340ff38-2a93-4102-8a87-c026f0ab26f7 per-consumer assignment: 02.12.2021 17:25:41,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,544 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 912326ee-6960-44d6-b637-f6bf4b4fb4d9 per-consumer assignment: 02.12.2021 17:25:41,544 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 7f1cad50-1794-4112-bb49-b2fba405b328 per-consumer assignment: 02.12.2021 17:25:41,544 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Removing task 0_5 from staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:41,544 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d due to tasks changing ownership. 02.12.2021 17:25:41,544 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Requested client 46d9f289-0630-4b19-b7f2-c59e02299906 to schedule a followup rebalance 02.12.2021 17:25:41,544 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 46d9f289-0630-4b19-b7f2-c59e02299906 per-consumer assignment: 02.12.2021 17:25:41,545 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 8f66a158-1279-467f-a02e-ff9e9f38632b per-consumer assignment: 02.12.2021 17:25:41,545 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 63f94986-6103-497f-970a-6fcbf19c32a5 per-consumer assignment: 02.12.2021 17:25:41,545 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f per-consumer assignment: 02.12.2021 17:25:41,545 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 941317db-575e-41d6-ace9-ee09236fae90 per-consumer assignment: 02.12.2021 17:25:41,545 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:41,545 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Finished assignment for group at generation 59: {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-0], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-2], userDataSize=48), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-7], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-3], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-1], userDataSize=48), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-6], userDataSize=48), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-4], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-8], userDataSize=48), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=Assignment(partitions=[], userDataSize=40)} 02.12.2021 17:25:41,546 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending leader SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 4, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:41,547 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953677, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:41,547 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953687, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:41,547 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1275) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]}]} 02.12.2021 17:25:41,551 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 40 ms and fetched 10 records 02.12.2021 17:25:41,558 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:41,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,558 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:41,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 24 iterations. 02.12.2021 17:25:41,562 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239882ms for 240000ms of delay, sleeping for 128ms 02.12.2021 17:25:41,572 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=669): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,572 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,572 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:41,572 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=670): org.apache.kafka.common.requests.HeartbeatResponse@627bc268 02.12.2021 17:25:41,572 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:41,572 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:41,572 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:41,572 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,569 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=669): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,573 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=670): org.apache.kafka.common.requests.HeartbeatResponse@227682f2 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:41,574 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:41,572 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,574 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,574 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,574 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:41,574 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,574 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from RUNNING to REBALANCING 02.12.2021 17:25:41,575 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:41,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 469 02.12.2021 17:25:41,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 467 02.12.2021 17:25:41,574 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:41,576 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,577 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:41,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 454 02.12.2021 17:25:41,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 443 02.12.2021 17:25:41,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:41,605 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 109 02.12.2021 17:25:41,604 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:41,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 295 02.12.2021 17:25:41,625 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:41,651 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1275): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,654 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,654 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:41,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 366 02.12.2021 17:25:41,692 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239560ms for 240000ms of delay, sleeping for 450ms 02.12.2021 17:25:41,683 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 343 02.12.2021 17:25:41,683 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:41,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 9 02.12.2021 17:25:41,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:41,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,723 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,724 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,724 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:41,726 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:41,726 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:41,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 265 02.12.2021 17:25:41,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:41,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 232 02.12.2021 17:25:41,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:41,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:41,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:41,878 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 165 02.12.2021 17:25:41,902 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 127 02.12.2021 17:25:41,902 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:41,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:41,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,923 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,926 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,926 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:41,926 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=962) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:41,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,928 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:41,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:41,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,932 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,932 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 64 02.12.2021 17:25:41,990 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:42,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,000 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,996 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59869ms for 60000ms of delay, sleeping for 141ms 02.12.2021 17:25:42,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,008 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 18 02.12.2021 17:25:42,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:42,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:42,030 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:42,030 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:42,030 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:42,030 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:42,030 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:42,030 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:42,030 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:42,030 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,032 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:42,026 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=962): org.apache.kafka.common.requests.HeartbeatResponse@4497c3ef 02.12.2021 17:25:42,037 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:42,031 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:42,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:42,030 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:42,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:42,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:42,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:42,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,050 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:42,054 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:42,054 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from REBALANCING to RUNNING 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,054 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 393 02.12.2021 17:25:42,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:42,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:42,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 287 02.12.2021 17:25:42,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 4 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:42,144 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1251] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Preparing to read 74677 bytes of data for partition redacted.Cloud.Message.Callback.Retry1-8 with offset FetchPosition{offset=1132519, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null)], epoch=absent}} 02.12.2021 17:25:42,144 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1275] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating high watermark for partition redacted.Cloud.Message.Callback.Retry1-8 to 1132683 02.12.2021 17:25:42,144 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1280] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating log start offset for partition redacted.Cloud.Message.Callback.Retry1-8 to 0 02.12.2021 17:25:42,144 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1285] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating last stable offset for partition redacted.Cloud.Message.Callback.Retry1-8 to 1132683 02.12.2021 17:25:42,144 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132519, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:42,144 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132529, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:42,144 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:42,145 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59333ms for 60000ms of delay, sleeping for 677ms 02.12.2021 17:25:42,145 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239949ms for 240000ms of delay, sleeping for 61ms 02.12.2021 17:25:42,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,208 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239988ms for 240000ms of delay, sleeping for 22ms 02.12.2021 17:25:42,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:42,224 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:42,224 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,224 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,224 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,224 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,224 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,225 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,231 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:42,233 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:42,233 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239498ms for 240000ms of delay, sleeping for 512ms 02.12.2021 17:25:42,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 186 02.12.2021 17:25:42,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:42,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:42,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:42,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:42,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:42,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 86 02.12.2021 17:25:42,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:42,410 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:42,425 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:42,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:42,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,429 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,434 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:42,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,455 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:42,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:42,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,502 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,503 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,526 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:42,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:42,537 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:42,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,540 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:42,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:42,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,550 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:42,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:42,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:42,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:42,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,555 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:42,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:42,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:42,640 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:42,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:42,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:42,726 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,726 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,726 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,726 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,726 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,726 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,726 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,726 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:42,740 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:42,743 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:42,747 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239825ms for 240000ms of delay, sleeping for 185ms 02.12.2021 17:25:42,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:42,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:42,828 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:42,831 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59622ms for 60000ms of delay, sleeping for 388ms 02.12.2021 17:25:42,833 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:42,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:42,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:42,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:42,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:42,930 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,929 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:42,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,934 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,936 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239772ms for 240000ms of delay, sleeping for 238ms 02.12.2021 17:25:42,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:42,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,939 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,940 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:42,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:43,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:43,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,008 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:43,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:43,034 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:43,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,043 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:43,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,051 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:43,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:43,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:43,051 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,056 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 490 02.12.2021 17:25:43,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:43,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:43,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:43,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 293 02.12.2021 17:25:43,144 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:43,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 390 02.12.2021 17:25:43,176 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239087ms for 240000ms of delay, sleeping for 923ms 02.12.2021 17:25:43,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:43,222 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59192ms for 60000ms of delay, sleeping for 818ms 02.12.2021 17:25:43,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:43,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,228 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:43,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:43,244 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:43,257 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 290 02.12.2021 17:25:43,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:43,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:43,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:43,344 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 90 02.12.2021 17:25:43,344 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:43,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 189 02.12.2021 17:25:43,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:43,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:43,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,434 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,443 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,443 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,446 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:43,459 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 88 02.12.2021 17:25:43,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:43,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,513 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,513 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:43,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:43,542 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:43,543 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,544 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:43,543 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=671) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:43,547 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:43,548 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,548 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=671) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:43,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:43,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,548 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=671): org.apache.kafka.common.requests.HeartbeatResponse@2e838255 02.12.2021 17:25:43,549 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:43,549 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=672) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:43,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 8 02.12.2021 17:25:43,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:43,550 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=671): org.apache.kafka.common.requests.HeartbeatResponse@5115ab20 02.12.2021 17:25:43,550 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:43,550 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:43,550 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:43,550 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:43,550 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,551 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=672) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:43,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:43,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:43,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,558 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:43,613 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:43,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:43,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:43,646 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:43,649 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:43,648 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:43,714 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:43,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:43,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,736 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,736 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:43,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:43,824 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:43,840 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:43,840 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:43,852 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:43,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:43,937 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:43,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:43,941 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,948 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,950 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 325 02.12.2021 17:25:43,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,953 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 317 02.12.2021 17:25:44,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:44,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,021 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,029 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,029 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,029 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,029 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 240 02.12.2021 17:25:44,042 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:44,041 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59161ms for 60000ms of delay, sleeping for 849ms 02.12.2021 17:25:44,051 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 224 02.12.2021 17:25:44,052 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:44,052 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,052 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,052 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,052 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,052 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,052 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,052 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,053 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:44,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:44,054 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 216 02.12.2021 17:25:44,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,059 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,054 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 24 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 25 iterations. 02.12.2021 17:25:44,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 25 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:44,103 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:44,104 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:44,104 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-8], userDataSize=48) 02.12.2021 17:25:44,104 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:44,104 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:330] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Skip resuming since state is RUNNING 02.12.2021 17:25:44,104 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:44,104 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:44,104 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from RUNNING to REBALANCING 02.12.2021 17:25:44,106 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:44,106 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,107 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 410 02.12.2021 17:25:44,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 138 02.12.2021 17:25:44,145 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:44,152 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 123 02.12.2021 17:25:44,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 115 02.12.2021 17:25:44,207 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,207 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,207 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 306 02.12.2021 17:25:44,231 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 38 02.12.2021 17:25:44,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:44,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,239 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,239 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 58 02.12.2021 17:25:44,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 22 02.12.2021 17:25:44,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 15 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,270 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:44,270 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=663) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 258 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,271 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 191 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,272 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=705) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null} 02.12.2021 17:25:44,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,272 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=705): org.apache.kafka.common.requests.HeartbeatResponse@66cfbdf4 02.12.2021 17:25:44,271 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=663): org.apache.kafka.common.requests.HeartbeatResponse@7f4807aa 02.12.2021 17:25:44,272 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:44,273 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 191 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:44,272 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:44,273 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:44,273 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:44,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 257 02.12.2021 17:25:44,273 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,275 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=706) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:44,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 254 02.12.2021 17:25:44,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,276 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,276 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 191 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 178 02.12.2021 17:25:44,276 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=681) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null} 02.12.2021 17:25:44,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 174 02.12.2021 17:25:44,278 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=681): org.apache.kafka.common.requests.HeartbeatResponse@31dd7604 02.12.2021 17:25:44,278 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:44,278 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 191 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:44,278 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:44,278 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:44,278 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,278 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=682) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:44,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 172 02.12.2021 17:25:44,297 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,298 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:44,298 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=663) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:44,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 442 02.12.2021 17:25:44,298 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=663): org.apache.kafka.common.requests.HeartbeatResponse@4bca6aa2 02.12.2021 17:25:44,298 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:44,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 441 02.12.2021 17:25:44,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 441 02.12.2021 17:25:44,319 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,319 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 194 02.12.2021 17:25:44,372 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:44,377 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:44,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 154 02.12.2021 17:25:44,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 341 02.12.2021 17:25:44,419 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,419 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30211ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30222ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 45 02.12.2021 17:25:44,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 240 02.12.2021 17:25:44,523 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,523 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,523 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:44,523 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1276) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null} 02.12.2021 17:25:44,524 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 81 02.12.2021 17:25:44,525 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1276): org.apache.kafka.common.requests.HeartbeatResponse@11bc445 02.12.2021 17:25:44,525 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:44,526 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,526 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:44,526 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:44,526 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:44,527 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:44,527 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1277) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:44,528 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:44,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:44,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:44,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,555 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30237ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,555 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:44,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:44,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:44,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,567 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30233ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,600 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 139 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:44,603 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:44,603 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from REBALANCING to RUNNING 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 25 iterations. 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 25 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:44,604 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953687, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:44,604 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953697, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:44,604 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 25 iterations. 02.12.2021 17:25:44,605 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239688ms for 240000ms of delay, sleeping for 322ms 02.12.2021 17:25:44,623 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:44,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:44,700 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 39 02.12.2021 17:25:44,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:44,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,743 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,748 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:44,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:44,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:44,916 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58800ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:44,938 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239913ms for 240000ms of delay, sleeping for 97ms 02.12.2021 17:25:44,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:44,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:44,949 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,952 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:44,952 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=963) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:44,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,952 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30717ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30728ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,029 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,037 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,040 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239818ms for 240000ms of delay, sleeping for 192ms 02.12.2021 17:25:45,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:45,052 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=963): org.apache.kafka.common.requests.HeartbeatResponse@799d216e 02.12.2021 17:25:45,053 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:45,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30740ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,058 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,067 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:45,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:45,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:45,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30751ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,086 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,137 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:45,152 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:45,237 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239290ms for 240000ms of delay, sleeping for 720ms 02.12.2021 17:25:45,238 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:45,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,250 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,251 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:45,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:45,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:45,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:45,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31224ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:45,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,474 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31244ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,479 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,479 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,479 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,544 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:45,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,545 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,546 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:45,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,558 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31241ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,573 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:45,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:45,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:45,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31252ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,587 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:45,652 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:45,691 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=672): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:45,691 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', members=[]) 02.12.2021 17:25:45,691 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:45,691 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:45,692 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:45,692 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=673) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null,assignments=[]} 02.12.2021 17:25:45,692 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 395 02.12.2021 17:25:45,691 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=672): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:45,701 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', members=[]) 02.12.2021 17:25:45,702 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:45,702 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:45,702 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:45,702 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=673) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:45,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 355 02.12.2021 17:25:45,749 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:45,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:45,755 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,755 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:45,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 392 02.12.2021 17:25:45,923 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59807ms delay out of 60000ms. Remaining 193ms. Executor queue at 0 02.12.2021 17:25:45,925 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59939ms for 60000ms of delay, sleeping for 71ms 02.12.2021 17:25:45,950 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:45,959 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239812ms for 240000ms of delay, sleeping for 198ms 02.12.2021 17:25:45,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,965 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:45,970 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31734ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,974 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,980 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31746ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,999 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59508ms for 60000ms of delay, sleeping for 502ms 02.12.2021 17:25:46,050 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:46,050 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,058 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:46,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,062 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31744ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,072 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 184 02.12.2021 17:25:46,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:46,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,093 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:46,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:46,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:46,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:46,093 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,098 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31768ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,104 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,160 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239996ms for 240000ms of delay, sleeping for 14ms 02.12.2021 17:25:46,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:46,176 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239781ms for 240000ms of delay, sleeping for 229ms 02.12.2021 17:25:46,186 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 379 02.12.2021 17:25:46,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,257 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,257 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 278 02.12.2021 17:25:46,357 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:46,389 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 175 02.12.2021 17:25:46,406 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239204ms for 240000ms of delay, sleeping for 806ms 02.12.2021 17:25:46,458 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32235ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32249ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,483 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,483 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 69 02.12.2021 17:25:46,504 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59432ms for 60000ms of delay, sleeping for 578ms 02.12.2021 17:25:46,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:46,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,566 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:46,594 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 521 ms and fetched 0 records 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32276ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,595 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32277ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,612 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:46,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:46,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,758 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,758 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:46,858 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:46,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:46,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:46,968 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:46,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32748ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:46,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32755ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:47,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:47,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,066 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 203 02.12.2021 17:25:47,084 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59724ms for 60000ms of delay, sleeping for 286ms 02.12.2021 17:25:47,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,095 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:47,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:47,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:47,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:47,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32778ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,096 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32778ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:47,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:47,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 25 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:47,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 26 iterations. 02.12.2021 17:25:47,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 26 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:47,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:47,213 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1277): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 42, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0])]) 02.12.2021 17:25:47,214 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -77, 64, -1, 56, 42, -109, 65, 2, -118, -121, -64, 38, -16, -85, 38, -9, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 70, -39, -14, -119, 6, 48, 75, 25, -73, -14, -59, -98, 2, 41, -103, 6, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 42, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -49, -35, 46, -67, -91, 11, 64, -19, -112, 118, 54, -49, 99, -2, 91, 66, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -113, 102, -95, 88, 18, 121, 70, 127, -96, 46, -1, -98, -97, 56, 99, 43, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 99, -7, 73, -122, 97, 3, 73, 127, -105, 10, 111, -53, -15, -100, 50, -91, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 98, -37, -94, -95, -98, 75, 74, -34, -67, -86, 25, 43, 40, 125, 10, 127, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 127, 28, -83, 80, 23, -108, 65, 18, -69, 73, -78, -5, -92, 5, -77, 40, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0])]) 02.12.2021 17:25:47,214 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:47,214 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:47,215 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@213604f, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@22ee2dca, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6cbb24ad, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6bb60271, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@723e1650, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@21ede5a0, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@14c6337c, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4f380e1f, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4d360d3e, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3b821dd, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2640af05, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@29070524, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@54a5b2c7, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@16d1c747, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@354a6a1f, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@24f567b7, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5dbfca64, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@454cc56b, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2330554f, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2cf75641, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@f384551, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2cfcae40, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@d471cbd, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1b14060a, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@564bdaf4, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1f0cf0d3, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2a9bb57b} 02.12.2021 17:25:47,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:47,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Constructed client metadata {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba, staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, b340ff38-2a93-4102-8a87-c026f0ab26f7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce, staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 912326ee-6960-44d6-b637-f6bf4b4fb4d9=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343, staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7f1cad50-1794-4112-bb49-b2fba405b328=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93, staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 46d9f289-0630-4b19-b7f2-c59e02299906=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b, staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 8f66a158-1279-467f-a02e-ff9e9f38632b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347, staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 63f94986-6103-497f-970a-6fcbf19c32a5=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e, staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab, staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 941317db-575e-41d6-ace9-ee09236fae90=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1, staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:47,218 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:47,218 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:47,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:47,219 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:47,219 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:47,219 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:47,219 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Queueing Call(callName=metadata, deadlineMs=1638462407219, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:47,219 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:47,219 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462407219, tries=0, nextAllowedTryMs=0)] at 1638462347219 02.12.2021 17:25:47,219 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Metadata is ready to use. 02.12.2021 17:25:47,219 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Found least loaded node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) connected with no in-flight requests 02.12.2021 17:25:47,219 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Assigned Call(callName=metadata, deadlineMs=1638462407219, tries=0, nextAllowedTryMs=0) to node kafka-3.data.sta.redacted:9093 (id: 2 rack: null) 02.12.2021 17:25:47,219 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_0]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:47,219 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-3.data.sta.redacted:9093 (id: 2 rack: null). correlationId=50 02.12.2021 17:25:47,220 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=50) and timeout 30000 to node 2: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:47,220 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=60000) 02.12.2021 17:25:47,220 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:47,220 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462347220 02.12.2021 17:25:47,220 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=59999) 02.12.2021 17:25:47,220 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Received METADATA response from node 2 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin, correlationId=50): org.apache.kafka.common.requests.MetadataResponse@57ad0d5d 02.12.2021 17:25:47,220 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:47,221 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Call(callName=metadata, deadlineMs=1638462407219, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:47,221 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Trying to choose nodes for [] at 1638462347220 02.12.2021 17:25:47,221 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-admin] Entering KafkaClient#poll(timeout=183862) 02.12.2021 17:25:47,222 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {8f66a158-1279-467f-a02e-ff9e9f38632b=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_0]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 912326ee-6960-44d6-b637-f6bf4b4fb4d9=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 941317db-575e-41d6-ace9-ee09236fae90=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], b340ff38-2a93-4102-8a87-c026f0ab26f7=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_3]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], cfdd2ebd-a50b-40ed-9076-36cf63fe5b42=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 46d9f289-0630-4b19-b7f2-c59e02299906=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 63f94986-6103-497f-970a-6fcbf19c32a5=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7f1cad50-1794-4112-bb49-b2fba405b328=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:47,222 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client cfdd2ebd-a50b-40ed-9076-36cf63fe5b42 per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client b340ff38-2a93-4102-8a87-c026f0ab26f7 per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 912326ee-6960-44d6-b637-f6bf4b4fb4d9 per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 7f1cad50-1794-4112-bb49-b2fba405b328 per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 46d9f289-0630-4b19-b7f2-c59e02299906 per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 8f66a158-1279-467f-a02e-ff9e9f38632b per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 63f94986-6103-497f-970a-6fcbf19c32a5 per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 62dba2a1-9e4b-4ade-bdaa-192b287d0a7f per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Client 941317db-575e-41d6-ace9-ee09236fae90 per-consumer assignment: 02.12.2021 17:25:47,223 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:997] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] Finished stable assignment of tasks, no followup rebalances required. 02.12.2021 17:25:47,224 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Finished assignment for group at generation 60: {staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-5], userDataSize=48), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-0], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-2], userDataSize=48), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-7], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-3], userDataSize=48), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-1], userDataSize=48), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-6], userDataSize=48), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-4], userDataSize=48), staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry3-8], userDataSize=48), staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025=Assignment(partitions=[], userDataSize=40)} 02.12.2021 17:25:47,224 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending leader SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 4, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:47,224 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953697, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:47,224 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953707, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:47,224 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1278) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-1-consumer-df0ca142-7c5f-42df-a2fa-3a7c6d6ad58d,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-3-consumer-630bc01b-e4b6-4c91-b7a5-b7d21f0c6b21,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-1-consumer-3f59373d-5dd8-4a5e-a879-3560c28615c4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-1-consumer-3b73301e-0e66-41ad-8964-3d4636ce2da3,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-46d9f289-0630-4b19-b7f2-c59e02299906-StreamThread-2-consumer-b91c36b8-6904-4a23-8ea2-f0f96ff0557b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-2-consumer-a0758338-fe5a-4015-a623-91904ee885ba,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-1-consumer-61727c3e-fad3-4e13-9fa1-1d2c6005d78c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-3-consumer-33faa43d-6f49-4cb9-85b0-f71e8782ff0b,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-1-consumer-70cc5830-9c35-4912-a371-6875ad664452,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-3-consumer-e6f2aeff-dff7-4070-a5ab-67dfff58a477,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-cfdd2ebd-a50b-40ed-9076-36cf63fe5b42-StreamThread-1-consumer-cb380aac-bfd6-4134-ab9c-84a470a80f16,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-3-consumer-b69830de-63f4-4cf6-8c6f-98ba24e81c7a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-2-consumer-4dedc8fc-4520-4fe9-bb56-cae5d927896e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-b340ff38-2a93-4102-8a87-c026f0ab26f7-StreamThread-2-consumer-b9ffa9f5-f1e3-41dd-abf8-432da904d3ce,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-3-consumer-3caf9825-5daf-4896-962d-7244548e6164,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-8f66a158-1279-467f-a02e-ff9e9f38632b-StreamThread-2-consumer-2b5da015-c3ea-4fdf-aabc-d7873066a347,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-62dba2a1-9e4b-4ade-bdaa-192b287d0a7f-StreamThread-2-consumer-a3465277-c427-47b7-a9f0-3cc77aa4baab,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry3-63f94986-6103-497f-970a-6fcbf19c32a5-StreamThread-3-consumer-6c62d289-f8c1-4884-9912-de494b3ab025,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]}]} 02.12.2021 17:25:47,224 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 11 ms and fetched 10 records 02.12.2021 17:25:47,225 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:47,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,225 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:47,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 26 iterations. 02.12.2021 17:25:47,227 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239382ms for 240000ms of delay, sleeping for 628ms 02.12.2021 17:25:47,240 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=673): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,240 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,240 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1', protocol='stream'} 02.12.2021 17:25:47,240 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=673): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,241 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 02.12.2021 17:25:47,241 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,241 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93', protocol='stream'} 02.12.2021 17:25:47,241 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:47,241 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:47,241 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 02.12.2021 17:25:47,241 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:47,241 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:47,241 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:47,241 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:47,241 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from RUNNING to REBALANCING 02.12.2021 17:25:47,241 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:47,241 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:47,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 356 02.12.2021 17:25:47,242 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:47,242 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:47,242 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:47,242 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:47,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 372 02.12.2021 17:25:47,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,259 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 38 02.12.2021 17:25:47,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 2 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,272 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:47,272 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:47,272 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=664) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:47,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:47,273 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=664): org.apache.kafka.common.requests.HeartbeatResponse@26485244 02.12.2021 17:25:47,273 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:47,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:47,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:47,297 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:47,297 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:47,298 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=664) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:47,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 462 02.12.2021 17:25:47,298 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=664): org.apache.kafka.common.requests.HeartbeatResponse@56f4075c 02.12.2021 17:25:47,298 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:47,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 461 02.12.2021 17:25:47,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 461 02.12.2021 17:25:47,325 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1278): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,325 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,325 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', protocol='stream'} 02.12.2021 17:25:47,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 254 02.12.2021 17:25:47,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 271 02.12.2021 17:25:47,371 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59264ms for 60000ms of delay, sleeping for 746ms 02.12.2021 17:25:47,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:47,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 361 02.12.2021 17:25:47,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 154 02.12.2021 17:25:47,443 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 170 02.12.2021 17:25:47,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:47,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:47,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,488 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33249ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,491 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:47,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,497 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33264ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,500 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 259 02.12.2021 17:25:47,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 70 02.12.2021 17:25:47,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 53 02.12.2021 17:25:47,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:47,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,567 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:47,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:47,598 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33280ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,598 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:47,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:47,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:47,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:47,602 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,601 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 158 02.12.2021 17:25:47,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:47,614 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:47,614 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:47,615 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:47,615 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:47,615 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:47,615 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:47,615 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:47,615 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:47,615 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from REBALANCING to RUNNING 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33285ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,620 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:47,699 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:47,702 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 57 02.12.2021 17:25:47,720 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:47,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,760 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:47,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:47,799 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:47,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:47,860 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239013ms for 240000ms of delay, sleeping for 997ms 02.12.2021 17:25:47,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:47,876 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:47,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:47,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:47,959 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:47,961 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:47,961 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=964) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:47,974 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:47,981 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:47,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33755ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33765ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,999 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,002 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:48,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:48,062 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=964): org.apache.kafka.common.requests.HeartbeatResponse@786de3f0 02.12.2021 17:25:48,065 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:48,074 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:48,074 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,074 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,076 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:48,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:48,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:48,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,103 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,109 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:48,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:48,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:48,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,121 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132529, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:48,122 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132539, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:48,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 4 ms and fetched 10 records 02.12.2021 17:25:48,122 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:48,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,122 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:48,124 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59404ms for 60000ms of delay, sleeping for 606ms 02.12.2021 17:25:48,121 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,129 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,132 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:48,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:48,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:48,215 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 396 02.12.2021 17:25:48,233 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:48,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:48,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,270 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:48,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:48,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:48,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:48,384 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:48,415 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:48,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:48,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:48,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 34256ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 34266ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:48,521 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:48,533 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:48,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:48,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:48,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,582 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,611 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:48,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,611 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:48,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,612 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,612 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:48,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:48,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,637 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 52 02.12.2021 17:25:48,672 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:48,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:48,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,691 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:48,692 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:48,692 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=674) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:48,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,692 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 447 02.12.2021 17:25:48,693 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=674): org.apache.kafka.common.requests.HeartbeatResponse@6779e7e7 02.12.2021 17:25:48,693 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:48,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 446 02.12.2021 17:25:48,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 446 02.12.2021 17:25:48,702 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:48,702 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:48,702 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=674) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:48,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 410 02.12.2021 17:25:48,703 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=674): org.apache.kafka.common.requests.HeartbeatResponse@33975a44 02.12.2021 17:25:48,703 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:48,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 409 02.12.2021 17:25:48,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 409 02.12.2021 17:25:48,731 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59963ms for 60000ms of delay, sleeping for 47ms 02.12.2021 17:25:48,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:48,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,770 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,779 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59557ms for 60000ms of delay, sleeping for 453ms 02.12.2021 17:25:48,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:48,794 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 345 02.12.2021 17:25:48,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 308 02.12.2021 17:25:48,859 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239957ms for 240000ms of delay, sleeping for 53ms 02.12.2021 17:25:48,872 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:48,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:48,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 244 02.12.2021 17:25:48,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 205 02.12.2021 17:25:48,923 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239705ms for 240000ms of delay, sleeping for 305ms 02.12.2021 17:25:48,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:48,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 144 02.12.2021 17:25:48,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 34757ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,998 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:49,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,000 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 34767ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 103 02.12.2021 17:25:49,077 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:49,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:49,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,091 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,091 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 44 02.12.2021 17:25:49,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,112 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:49,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:49,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:49,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:49,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:49,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,118 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:49,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,140 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,140 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,177 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:49,192 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:49,219 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:49,230 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239281ms for 240000ms of delay, sleeping for 729ms 02.12.2021 17:25:49,233 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59394ms for 60000ms of delay, sleeping for 616ms 02.12.2021 17:25:49,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:49,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,272 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:49,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:49,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:49,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:49,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:49,420 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:49,441 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:49,473 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:49,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:49,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,499 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 35259ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,507 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 35273ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,508 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,521 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:49,541 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:49,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:49,592 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,592 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,593 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,613 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:49,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:49,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:49,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:49,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,621 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,621 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:49,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,641 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:49,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:49,721 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:49,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:49,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:49,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,778 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,779 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,794 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:49,822 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:49,843 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:49,863 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59971ms for 60000ms of delay, sleeping for 39ms 02.12.2021 17:25:49,879 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:49,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:49,923 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:49,920 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59798ms for 60000ms of delay, sleeping for 212ms 02.12.2021 17:25:49,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:49,962 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239506ms for 240000ms of delay, sleeping for 504ms 02.12.2021 17:25:49,986 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:49,998 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:50,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 35761ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:50,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,009 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 35784ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,019 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:50,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:50,056 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=682): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:50,056 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:50,056 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=706): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:50,057 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:50,057 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:50,057 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:50,057 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:50,057 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=707) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:50,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 462 02.12.2021 17:25:50,056 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1039): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:50,058 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1040) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,assignments=[]} 02.12.2021 17:25:50,058 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 56 02.12.2021 17:25:50,057 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:50,060 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:50,061 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:50,061 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=683) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,assignments=[]} 02.12.2021 17:25:50,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 440 02.12.2021 17:25:50,088 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:50,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,094 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 177 02.12.2021 17:25:50,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:50,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,117 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:50,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:50,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:50,121 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:50,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,127 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,139 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59951ms for 60000ms of delay, sleeping for 59ms 02.12.2021 17:25:50,147 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:50,147 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,147 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,148 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,160 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1040): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:50,160 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:50,160 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:50,160 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 192 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:50,160 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:50,160 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:50,160 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:50,161 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:50,161 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,161 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from RUNNING to REBALANCING 02.12.2021 17:25:50,161 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:50,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 461 02.12.2021 17:25:50,160 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=707): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,163 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,160 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=683): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,163 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:50,163 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 192 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:50,164 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:50,164 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:50,163 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,164 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=192, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:50,164 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 192 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:50,164 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:50,164 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:50,164 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:50,164 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:50,164 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,164 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:50,165 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:50,165 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,165 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:50,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 356 02.12.2021 17:25:50,165 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:50,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 338 02.12.2021 17:25:50,188 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:50,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 77 02.12.2021 17:25:50,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:50,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:50,200 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132539, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:50,200 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132549, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:50,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:50,200 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:50,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,200 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:50,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:50,200 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59939ms for 60000ms of delay, sleeping for 71ms 02.12.2021 17:25:50,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:50,233 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,233 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:50,233 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1279) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null} 02.12.2021 17:25:50,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:50,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 360 02.12.2021 17:25:50,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 254 02.12.2021 17:25:50,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 233 02.12.2021 17:25:50,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,272 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,273 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:50,273 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=665) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:50,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 323 02.12.2021 17:25:50,273 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=665): org.apache.kafka.common.requests.HeartbeatResponse@52392f14 02.12.2021 17:25:50,273 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:50,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 322 02.12.2021 17:25:50,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 321 02.12.2021 17:25:50,274 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59424ms for 60000ms of delay, sleeping for 586ms 02.12.2021 17:25:50,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,279 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,280 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,280 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,280 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,280 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 17 02.12.2021 17:25:50,297 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,297 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:50,297 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=665) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:50,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 483 02.12.2021 17:25:50,297 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=665): org.apache.kafka.common.requests.HeartbeatResponse@2c8c3cc9 02.12.2021 17:25:50,298 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:50,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 482 02.12.2021 17:25:50,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 482 02.12.2021 17:25:50,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:50,333 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1279): org.apache.kafka.common.requests.HeartbeatResponse@50e7e9df 02.12.2021 17:25:50,333 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:50,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:50,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 260 02.12.2021 17:25:50,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 153 02.12.2021 17:25:50,369 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 132 02.12.2021 17:25:50,375 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 220 02.12.2021 17:25:50,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 382 02.12.2021 17:25:50,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:50,449 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:50,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 159 02.12.2021 17:25:50,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 26 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 27 iterations. 02.12.2021 17:25:50,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 27 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:50,469 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 02.12.2021 17:25:50,469 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:50,469 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-8], userDataSize=48) 02.12.2021 17:25:50,469 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:50,469 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:330] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Skip resuming since state is RUNNING 02.12.2021 17:25:50,469 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:50,470 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,470 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from RUNNING to REBALANCING 02.12.2021 17:25:50,470 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:50,470 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,470 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,470 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 49 02.12.2021 17:25:50,473 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 29 02.12.2021 17:25:50,476 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 119 02.12.2021 17:25:50,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 281 02.12.2021 17:25:50,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,502 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:50,503 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,503 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,503 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:50,503 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:50,504 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,504 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,504 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:50,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,504 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Committing all active tasks [] and standby tasks [] since 36265ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:50,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,506 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,520 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:50,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,521 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Committing all active tasks [] and standby tasks [] since 36289ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:50,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:50,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:50,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 59 02.12.2021 17:25:50,570 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,570 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:50,576 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 19 02.12.2021 17:25:50,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,597 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,597 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:50,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 181 02.12.2021 17:25:50,606 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:50,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:50,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,623 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:50,623 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,624 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,624 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from REBALANCING to RUNNING 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,624 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:50,624 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 192 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:50,624 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:50,625 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:50,625 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:50,625 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1041) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:50,625 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:50,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:50,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,629 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,650 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,663 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:50,671 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,672 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:50,698 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:50,701 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:50,706 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:50,724 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:50,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:50,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:50,774 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,774 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:50,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:50,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:50,800 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:50,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:50,832 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:50,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:50,862 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59666ms for 60000ms of delay, sleeping for 344ms 02.12.2021 17:25:50,879 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,880 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:50,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:50,925 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 81 02.12.2021 17:25:50,929 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:50,928 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 169 02.12.2021 17:25:50,932 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:50,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:50,974 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=965) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:50,987 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-8 because it is paused 02.12.2021 17:25:50,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 519 ms and fetched 0 records 02.12.2021 17:25:50,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,988 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:50,988 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,988 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,988 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [redacted.Cloud.Message.Callback.Retry3-8] 02.12.2021 17:25:50,988 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:50,988 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,989 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,989 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328] State transition from REBALANCING to RUNNING 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 27 iterations. 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 27 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:50,989 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953707, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:50,989 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953717, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:50,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:50,990 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:50,990 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,990 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:50,990 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,990 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 27 iterations. 02.12.2021 17:25:50,991 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239777ms for 240000ms of delay, sleeping for 233ms 02.12.2021 17:25:50,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:51,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:51,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,012 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,016 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,026 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,035 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:51,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 66 02.12.2021 17:25:51,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:51,078 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=965): org.apache.kafka.common.requests.HeartbeatResponse@74617919 02.12.2021 17:25:51,079 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:51,092 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:51,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:51,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,102 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,102 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,122 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 394 02.12.2021 17:25:51,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,125 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:51,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:51,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:51,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:51,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:51,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,130 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,130 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:51,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,151 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,151 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:51,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 264 02.12.2021 17:25:51,251 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:51,230 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:51,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:51,438 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239783ms for 240000ms of delay, sleeping for 227ms 02.12.2021 17:25:51,203 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:51,192 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:51,459 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59577ms for 60000ms of delay, sleeping for 433ms 02.12.2021 17:25:51,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 747 ms and fetched 0 records 02.12.2021 17:25:51,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,530 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,537 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 525 ms and fetched 0 records 02.12.2021 17:25:51,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:51,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 63 02.12.2021 17:25:51,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:51,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 113 02.12.2021 17:25:51,541 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,604 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,605 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:51,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:51,630 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,631 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,632 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 71 02.12.2021 17:25:51,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,633 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:51,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:51,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:51,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:51,634 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:51,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 10 02.12.2021 17:25:51,651 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:51,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:51,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,658 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,659 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,659 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,659 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,659 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,659 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,659 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 32 02.12.2021 17:25:51,673 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239699ms for 240000ms of delay, sleeping for 311ms 02.12.2021 17:25:51,691 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:51,692 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:51,692 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=675) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:51,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,692 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 467 02.12.2021 17:25:51,693 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=675): org.apache.kafka.common.requests.HeartbeatResponse@510f91ed 02.12.2021 17:25:51,693 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:51,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 466 02.12.2021 17:25:51,703 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:51,703 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:51,703 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=675) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:51,697 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 462 02.12.2021 17:25:51,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 428 02.12.2021 17:25:51,707 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=675): org.apache.kafka.common.requests.HeartbeatResponse@2f4bbb81 02.12.2021 17:25:51,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:51,717 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:51,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 414 02.12.2021 17:25:51,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 412 02.12.2021 17:25:51,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:51,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 295 02.12.2021 17:25:51,752 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:51,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 355 02.12.2021 17:25:51,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:51,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:51,818 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 287 02.12.2021 17:25:51,853 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:51,852 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 190 02.12.2021 17:25:51,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 252 02.12.2021 17:25:51,909 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59576ms for 60000ms of delay, sleeping for 434ms 02.12.2021 17:25:51,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:51,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 164 02.12.2021 17:25:51,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 190 02.12.2021 17:25:51,953 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:51,957 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 85 02.12.2021 17:25:51,991 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239059ms for 240000ms of delay, sleeping for 951ms 02.12.2021 17:25:52,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 150 02.12.2021 17:25:52,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,036 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,050 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,050 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,050 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 81 02.12.2021 17:25:52,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:52,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 516 ms and fetched 0 records 02.12.2021 17:25:52,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,058 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:52,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,065 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,077 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 50 02.12.2021 17:25:52,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:52,121 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,121 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,137 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:52,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:52,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:52,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:52,137 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:52,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,139 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:52,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:52,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,171 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 393 02.12.2021 17:25:52,180 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:52,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:52,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:52,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:52,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 286 02.12.2021 17:25:52,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 289 02.12.2021 17:25:52,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:52,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:52,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:52,368 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59420ms for 60000ms of delay, sleeping for 590ms 02.12.2021 17:25:52,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:52,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 185 02.12.2021 17:25:52,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 189 02.12.2021 17:25:52,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:52,446 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:52,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:52,476 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 83 02.12.2021 17:25:52,472 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:52,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 79 02.12.2021 17:25:52,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:52,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:52,551 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,553 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,560 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 497 02.12.2021 17:25:52,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:52,579 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:52,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,580 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,581 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 479 02.12.2021 17:25:52,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,625 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,625 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,638 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:52,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:52,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:52,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:52,638 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,640 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:52,660 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:52,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,671 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,671 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,681 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 379 02.12.2021 17:25:52,726 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:52,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,753 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:52,761 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:52,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 278 02.12.2021 17:25:52,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:52,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:52,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:52,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:52,872 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:52,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 178 02.12.2021 17:25:52,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:52,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:52,947 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239256ms for 240000ms of delay, sleeping for 754ms 02.12.2021 17:25:52,954 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:52,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:52,975 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59988ms for 60000ms of delay, sleeping for 22ms 02.12.2021 17:25:52,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:52,984 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 76 02.12.2021 17:25:53,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:53,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:53,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:53,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:53,010 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132549, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:53,010 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132559, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:53,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:53,014 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:53,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,014 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:53,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:53,014 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59766ms for 60000ms of delay, sleeping for 244ms 02.12.2021 17:25:53,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:53,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:53,059 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 192 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=708) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null} 02.12.2021 17:25:53,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:53,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,063 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,062 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,064 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 192 and member id staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,062 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:53,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,065 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,065 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=684) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null} 02.12.2021 17:25:53,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 233 02.12.2021 17:25:53,067 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=684): org.apache.kafka.common.requests.HeartbeatResponse@7406629 02.12.2021 17:25:53,067 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:53,067 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 192 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:53,067 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:53,067 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:53,067 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,067 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=685) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:53,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 13 02.12.2021 17:25:53,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,069 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=708): org.apache.kafka.common.requests.HeartbeatResponse@2f3db0d1 02.12.2021 17:25:53,072 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:53,076 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 192 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:53,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:53,078 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:53,080 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,081 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,082 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=709) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:53,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 484 02.12.2021 17:25:53,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:53,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,131 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 141 02.12.2021 17:25:53,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,139 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:53,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:53,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:53,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:53,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:53,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,145 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,145 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,161 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:53,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 131 02.12.2021 17:25:53,167 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:53,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,172 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,232 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 40 02.12.2021 17:25:53,245 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:53,264 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59193ms for 60000ms of delay, sleeping for 817ms 02.12.2021 17:25:53,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 30 02.12.2021 17:25:53,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,272 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,272 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:53,272 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=666) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:53,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:53,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 359 02.12.2021 17:25:53,273 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=666): org.apache.kafka.common.requests.HeartbeatResponse@12add194 02.12.2021 17:25:53,273 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:53,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 358 02.12.2021 17:25:53,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 358 02.12.2021 17:25:53,290 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,290 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:53,290 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1280) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null} 02.12.2021 17:25:53,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,302 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,303 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:53,303 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=666) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:53,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 262 02.12.2021 17:25:53,305 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=666): org.apache.kafka.common.requests.HeartbeatResponse@76e3e6f0 02.12.2021 17:25:53,313 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:53,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 251 02.12.2021 17:25:53,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 251 02.12.2021 17:25:53,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:53,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:53,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 258 02.12.2021 17:25:53,391 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1280): org.apache.kafka.common.requests.HeartbeatResponse@4b56cd23 02.12.2021 17:25:53,391 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:53,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 150 02.12.2021 17:25:53,447 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:53,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 157 02.12.2021 17:25:53,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:53,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 45 02.12.2021 17:25:53,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:53,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:53,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,569 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:53,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 56 02.12.2021 17:25:53,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:53,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,583 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,632 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,632 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:53,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,640 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:53,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:53,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:53,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:53,640 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:53,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,655 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:53,674 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,702 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239735ms for 240000ms of delay, sleeping for 275ms 02.12.2021 17:25:53,732 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:53,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:53,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:53,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:53,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:53,858 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:53,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:53,879 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:53,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:53,960 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:53,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:53,986 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,990 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:53,990 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:53,990 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239859ms for 240000ms of delay, sleeping for 151ms 02.12.2021 17:25:53,994 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=966) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:54,036 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:54,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:54,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:54,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:54,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,078 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,078 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,073 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,085 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:54,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,092 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59576ms for 60000ms of delay, sleeping for 434ms 02.12.2021 17:25:54,098 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=966): org.apache.kafka.common.requests.HeartbeatResponse@e1354c1 02.12.2021 17:25:54,098 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:54,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:54,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,139 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,141 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:54,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:54,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:54,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:54,147 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 27 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,147 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,147 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 28 iterations. 02.12.2021 17:25:54,147 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 28 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:54,148 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953717, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:54,148 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953727, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:54,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:54,149 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:54,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,149 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:54,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 28 iterations. 02.12.2021 17:25:54,152 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239993ms for 240000ms of delay, sleeping for 17ms 02.12.2021 17:25:54,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:54,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,162 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,162 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:54,173 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238796ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:54,178 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:54,179 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:54,179 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,179 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,179 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,179 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,179 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,179 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,179 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,180 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:54,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:54,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:54,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:54,280 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:54,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:54,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:54,379 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:54,381 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:54,440 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:54,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:54,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:54,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:54,538 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59535ms for 60000ms of delay, sleeping for 475ms 02.12.2021 17:25:54,541 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:54,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:54,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:54,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,581 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,581 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:54,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,582 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:54,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:54,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,591 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,592 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,640 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,640 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,642 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:54,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:54,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:54,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,663 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,664 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 39 02.12.2021 17:25:54,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:54,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:54,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,684 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 7 02.12.2021 17:25:54,692 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,692 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:54,692 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=676) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:54,692 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 492 02.12.2021 17:25:54,692 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=676): org.apache.kafka.common.requests.HeartbeatResponse@47f8f82c 02.12.2021 17:25:54,693 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:54,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 491 02.12.2021 17:25:54,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 490 02.12.2021 17:25:54,703 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,703 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:54,703 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=676) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:54,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 461 02.12.2021 17:25:54,703 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=676): org.apache.kafka.common.requests.HeartbeatResponse@131f5482 02.12.2021 17:25:54,704 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:54,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 460 02.12.2021 17:25:54,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 460 02.12.2021 17:25:54,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:54,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:54,794 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 390 02.12.2021 17:25:54,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 360 02.12.2021 17:25:54,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:54,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:54,894 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 290 02.12.2021 17:25:54,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 260 02.12.2021 17:25:54,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:54,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:54,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 189 02.12.2021 17:25:55,006 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 158 02.12.2021 17:25:55,023 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59344ms for 60000ms of delay, sleeping for 666ms 02.12.2021 17:25:55,042 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:55,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,083 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:55,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,084 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,093 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,093 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,099 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 85 02.12.2021 17:25:55,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 49 02.12.2021 17:25:55,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:55,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,142 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:55,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,144 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:55,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:55,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:55,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:55,145 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,164 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,164 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,174 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239797ms delay out of 240000ms. Remaining 203ms. Executor queue at 0 02.12.2021 17:25:55,175 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239071ms for 240000ms of delay, sleeping for 939ms 02.12.2021 17:25:55,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:55,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,184 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,243 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:55,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:55,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:55,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:55,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:55,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:55,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:55,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:55,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:55,465 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:55,487 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:55,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 200 02.12.2021 17:25:55,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:55,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:55,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,586 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,588 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,591 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:55,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,594 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,597 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,646 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:55,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:55,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:55,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:55,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:55,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,648 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,648 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,646 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:55,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,669 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:55,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,689 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:55,697 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59795ms for 60000ms of delay, sleeping for 215ms 02.12.2021 17:25:55,749 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:55,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:55,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:55,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:55,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:55,872 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:55,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:55,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:55,918 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58951ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:55,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:55,973 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:56,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 187 02.12.2021 17:25:56,006 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:56,056 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:56,074 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:56,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,096 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,098 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,098 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,101 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,101 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 201 02.12.2021 17:25:56,104 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 86 02.12.2021 17:25:56,126 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239908ms for 240000ms of delay, sleeping for 102ms 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:56,151 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:56,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,159 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,159 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 113 02.12.2021 17:25:56,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,169 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:56,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,191 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,193 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,205 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:56,231 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239954ms for 240000ms of delay, sleeping for 56ms 02.12.2021 17:25:56,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 13 02.12.2021 17:25:56,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:56,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,273 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:56,273 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:56,273 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=667) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:56,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 386 02.12.2021 17:25:56,273 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=667): org.apache.kafka.common.requests.HeartbeatResponse@2de7f13c 02.12.2021 17:25:56,273 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:56,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 386 02.12.2021 17:25:56,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 385 02.12.2021 17:25:56,288 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239561ms for 240000ms of delay, sleeping for 449ms 02.12.2021 17:25:56,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:56,297 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:56,297 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:56,297 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1281) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null} 02.12.2021 17:25:56,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,302 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:56,302 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:56,302 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=667) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:56,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:56,303 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=667): org.apache.kafka.common.requests.HeartbeatResponse@6ee40df4 02.12.2021 17:25:56,303 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:56,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:56,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:56,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:56,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:56,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:56,397 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1281): org.apache.kafka.common.requests.HeartbeatResponse@416fc518 02.12.2021 17:25:56,400 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:56,403 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:56,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 200 02.12.2021 17:25:56,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:56,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:56,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:56,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:56,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:56,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,597 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,597 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:56,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,601 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,602 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,602 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:56,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,652 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:56,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:56,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:56,652 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:56,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,664 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,664 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:56,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:56,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,674 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:56,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,695 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:56,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 10 records with 28 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 29 iterations. 02.12.2021 17:25:56,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processed 0 records with 29 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,739 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:56,740 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=953727, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:56,741 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=953737, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-8 02.12.2021 17:25:56,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:56,742 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-8, new queue size is 10 02.12.2021 17:25:56,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,742 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:56,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1] Processing tasks with 29 iterations. 02.12.2021 17:25:56,744 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239846ms for 240000ms of delay, sleeping for 164ms 02.12.2021 17:25:56,765 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:56,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:56,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:56,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:56,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:56,878 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:56,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:56,905 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:56,920 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59953ms delay out of 60000ms. Remaining 47ms. Executor queue at 0 02.12.2021 17:25:56,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 10 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 18 iterations. 02.12.2021 17:25:56,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processed 0 records with 18 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:56,920 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1132559, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:56,920 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1132569, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-8 02.12.2021 17:25:56,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:56,921 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] task [0_8] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-8, new queue size is 10 02.12.2021 17:25:56,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,921 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:56,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1] Processing tasks with 18 iterations. 02.12.2021 17:25:56,921 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59861ms for 60000ms of delay, sleeping for 149ms 02.12.2021 17:25:56,923 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239039ms for 240000ms of delay, sleeping for 971ms 02.12.2021 17:25:56,968 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:56,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:57,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:57,002 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,004 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:57,004 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=967) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer-f8bb755e-5a3e-46f0-a05e-066b26cd1c1b,group_instance_id=null} 02.12.2021 17:25:57,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:57,069 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:57,072 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59781ms for 60000ms of delay, sleeping for 229ms 02.12.2021 17:25:57,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:57,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,098 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,098 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,102 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:57,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,103 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,103 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,107 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, correlationId=967): org.apache.kafka.common.requests.HeartbeatResponse@1e95923c 02.12.2021 17:25:57,107 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:57,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:57,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,108 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,152 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:57,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:57,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:57,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:57,152 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,165 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,176 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,176 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:57,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:57,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,198 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,198 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 494 02.12.2021 17:25:57,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:57,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:57,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:57,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 394 02.12.2021 17:25:57,303 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59607ms for 60000ms of delay, sleeping for 403ms 02.12.2021 17:25:57,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:57,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:57,378 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:57,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 293 02.12.2021 17:25:57,409 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:57,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:57,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:57,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:57,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:57,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:57,580 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:57,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,600 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,601 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 88 02.12.2021 17:25:57,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,606 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,614 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,615 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,653 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:57,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:57,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:57,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:57,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,667 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,667 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,667 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,667 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,667 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,667 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,667 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,678 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 25 02.12.2021 17:25:57,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,699 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:57,699 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=677) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer-2858c2cc-b143-46d2-a8b9-f92eee3a5d93,group_instance_id=null} 02.12.2021 17:25:57,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,699 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, correlationId=677): org.apache.kafka.common.requests.HeartbeatResponse@755b4bac 02.12.2021 17:25:57,700 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:57,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:57,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,700 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:57,704 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,704 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:57,706 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=677) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer-9ff2155f-d0ea-4b15-933a-9c22e3a244d1,group_instance_id=null} 02.12.2021 17:25:57,706 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 474 02.12.2021 17:25:57,707 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, correlationId=677): org.apache.kafka.common.requests.HeartbeatResponse@6f2fb16a 02.12.2021 17:25:57,707 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:57,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 471 02.12.2021 17:25:57,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 471 02.12.2021 17:25:57,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:57,714 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59747ms for 60000ms of delay, sleeping for 263ms 02.12.2021 17:25:57,715 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:57,768 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:57,808 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 370 02.12.2021 17:25:57,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 392 02.12.2021 17:25:57,815 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:57,873 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:57,910 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 268 02.12.2021 17:25:57,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 291 02.12.2021 17:25:57,913 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239730ms for 240000ms of delay, sleeping for 280ms 02.12.2021 17:25:57,917 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:57,973 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:58,001 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59023ms for 60000ms of delay, sleeping for 987ms 02.12.2021 17:25:58,010 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 168 02.12.2021 17:25:58,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 191 02.12.2021 17:25:58,029 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:58,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:58,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,109 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:58,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 66 02.12.2021 17:25:58,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:58,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:58,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,123 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,122 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,123 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,134 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,158 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:58,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:58,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:58,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:58,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:58,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,171 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:58,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,188 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,188 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,188 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,195 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239845ms for 240000ms of delay, sleeping for 165ms 02.12.2021 17:25:58,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:58,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,211 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,211 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,235 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:58,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:58,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:58,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:58,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:58,362 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239412ms for 240000ms of delay, sleeping for 598ms 02.12.2021 17:25:58,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:58,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:58,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:58,436 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:58,473 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:58,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:58,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:58,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:58,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:58,594 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:58,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:58,618 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:58,620 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,620 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,621 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,636 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,635 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:58,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,663 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:58,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:58,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:58,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:58,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,673 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30305ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:58,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:58,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,691 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:58,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,713 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,713 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,736 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:58,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:58,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:58,813 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:58,837 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:58,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:58,905 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:58,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 291 02.12.2021 17:25:58,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:58,963 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239753ms for 240000ms of delay, sleeping for 257ms 02.12.2021 17:25:58,976 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:58,991 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58875ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:59,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:59,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:59,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:59,076 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:59,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:59,121 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:59,121 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,121 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,121 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,122 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,126 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 87 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,137 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,137 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30501ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:59,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,138 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 164 02.12.2021 17:25:59,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,164 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:59,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:59,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:59,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:59,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:59,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,175 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:59,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:59,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,203 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,203 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,214 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,214 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,223 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239943ms for 240000ms of delay, sleeping for 67ms 02.12.2021 17:25:59,239 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:59,273 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:59,274 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:59,275 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=668) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer-dde93885-0a73-46d8-8f31-c2596f11a02e,group_instance_id=null} 02.12.2021 17:25:59,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 402 02.12.2021 17:25:59,276 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, correlationId=668): org.apache.kafka.common.requests.HeartbeatResponse@49595cb6 02.12.2021 17:25:59,277 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:59,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:59,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:59,281 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1041): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:59,281 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', members=[]) 02.12.2021 17:25:59,281 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:59,281 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:59,281 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=685): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:59,281 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:59,281 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=709): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:59,281 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1042) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=193,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,assignments=[]} 02.12.2021 17:25:59,281 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', members=[]) 02.12.2021 17:25:59,282 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:59,282 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:59,282 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:59,282 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=686) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=193,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,group_instance_id=null,assignments=[]} 02.12.2021 17:25:59,282 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', members=[]) 02.12.2021 17:25:59,283 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:59,283 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:59,283 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:59,283 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=710) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=193,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:59,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 355 02.12.2021 17:25:59,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 383 02.12.2021 17:25:59,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 339 02.12.2021 17:25:59,291 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239190ms for 240000ms of delay, sleeping for 820ms 02.12.2021 17:25:59,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:59,303 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:59,303 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:59,303 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=668) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer-91dfde1f-6cb9-4774-bf3e-f28c87630409,group_instance_id=null} 02.12.2021 17:25:59,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 335 02.12.2021 17:25:59,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:59,305 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, correlationId=668): org.apache.kafka.common.requests.HeartbeatResponse@18dc3993 02.12.2021 17:25:59,305 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:59,305 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 333 02.12.2021 17:25:59,305 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 333 02.12.2021 17:25:59,307 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:59,307 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:59,307 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1282) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692,group_instance_id=null} 02.12.2021 17:25:59,313 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1042): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:59,314 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:59,314 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, correlationId=686): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,314 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,314 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', protocol='stream'} 02.12.2021 17:25:59,314 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 193 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28 02.12.2021 17:25:59,314 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, correlationId=710): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,314 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,314 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:59,315 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:59,314 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', protocol='stream'} 02.12.2021 17:25:59,315 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=193, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', protocol='stream'} 02.12.2021 17:25:59,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:59,315 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 193 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e 02.12.2021 17:25:59,315 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:59,316 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:59,316 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:59,315 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 193 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:59,316 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:59,316 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:59,315 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:59,316 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:59,316 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:59,317 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:59,317 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:59,316 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:59,317 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:59,317 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from RUNNING to REBALANCING 02.12.2021 17:25:59,317 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:59,317 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:59,317 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:59,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 323 02.12.2021 17:25:59,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 308 02.12.2021 17:25:59,317 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:59,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 351 02.12.2021 17:25:59,377 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:59,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:59,405 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 233 02.12.2021 17:25:59,408 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, correlationId=1282): org.apache.kafka.common.requests.HeartbeatResponse@2c53aa02 02.12.2021 17:25:59,408 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:59,416 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:59,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 205 02.12.2021 17:25:59,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 220 02.12.2021 17:25:59,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 247 02.12.2021 17:25:59,477 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:59,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:59,506 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 132 02.12.2021 17:25:59,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:59,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 104 02.12.2021 17:25:59,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 119 02.12.2021 17:25:59,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 147 02.12.2021 17:25:59,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:59,608 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:59,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 31 02.12.2021 17:25:59,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 46 02.12.2021 17:25:59,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 18 02.12.2021 17:25:59,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 3 02.12.2021 17:25:59,617 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:59,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:59,624 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:59,638 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:59,638 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:59,639 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:59,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,644 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,646 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,646 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:59,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:59,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:59,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:59,666 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:59,667 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:59,667 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:59,669 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:59,669 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:59,669 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:59,669 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:59,670 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea] State transition from REBALANCING to RUNNING 02.12.2021 17:25:59,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:59,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processing tasks with 36 iterations. 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Processed 0 records with 36 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:59,675 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 193 and memberId staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa 02.12.2021 17:25:59,675 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:59,675 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, correlationId=1043) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:59,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:59,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,678 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:59,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,705 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,719 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:59,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:59,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:59,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:59,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:59,808 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:59,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:59,831 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:59,840 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:59,847 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:59,889 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:59,910 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:59,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:59,931 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:59,940 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:59,948 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:59,990 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-5342043b-e85b-472b-8357-e0dc8eb244f1-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:59,992 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59876ms delay out of 60000ms. Remaining 124ms. Executor queue at 0 02.12.2021 17:25:59,997 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59872ms for 60000ms of delay, sleeping for 138ms