02.12.2021 17:25:00,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:00,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,035 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,036 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119899ms for 120000ms of delay, sleeping for 111ms 02.12.2021 17:25:00,035 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,050 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:00,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:00,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:00,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:00,122 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59107ms delay out of 60000ms. Remaining 893ms. Executor queue at 0 02.12.2021 17:25:00,123 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59072ms for 60000ms of delay, sleeping for 938ms 02.12.2021 17:25:00,130 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239952ms for 240000ms of delay, sleeping for 58ms 02.12.2021 17:25:00,149 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119965ms for 120000ms of delay, sleeping for 45ms 02.12.2021 17:25:00,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:00,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:00,195 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239061ms for 240000ms of delay, sleeping for 949ms 02.12.2021 17:25:00,196 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119301ms for 120000ms of delay, sleeping for 709ms 02.12.2021 17:25:00,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,204 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,207 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,217 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:00,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,250 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,270 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 318 02.12.2021 17:25:00,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:00,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:00,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:00,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 218 02.12.2021 17:25:00,411 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:00,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:00,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,434 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,434 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:00,470 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 118 02.12.2021 17:25:00,512 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:00,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:00,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,517 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 69 02.12.2021 17:25:00,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:00,543 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,543 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,543 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,543 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,543 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,543 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,543 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:00,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 17 02.12.2021 17:25:00,587 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:00,587 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 185 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,587 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=685) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=185,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null} 02.12.2021 17:25:00,587 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 431 02.12.2021 17:25:00,588 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=685): org.apache.kafka.common.requests.HeartbeatResponse@3db43d4e 02.12.2021 17:25:00,588 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:00,588 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 185 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:00,588 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:00,588 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:00,588 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,588 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=686) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:00,588 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 430 02.12.2021 17:25:00,590 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:00,590 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 185 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,590 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=746) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=185,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null} 02.12.2021 17:25:00,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,591 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=746): org.apache.kafka.common.requests.HeartbeatResponse@7644cf90 02.12.2021 17:25:00,591 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:00,592 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 185 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:00,592 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:00,592 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:00,592 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,592 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=747) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:00,592 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 178 02.12.2021 17:25:00,612 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:00,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:00,687 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:00,692 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:00,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:00,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,712 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,712 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,712 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,713 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,752 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,752 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,757 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,771 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,772 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,823 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 390 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 58 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 104240ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 59 iterations. 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 59 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 104240ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:00,896 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 185 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,910 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1139) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=185,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null} 02.12.2021 17:25:00,910 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Rebalance failed. 02.12.2021 17:25:00,911 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:00,911 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:00,911 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 393 02.12.2021 17:25:00,911 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1140) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:00,913 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1139): org.apache.kafka.common.requests.HeartbeatResponse@46297825 02.12.2021 17:25:00,918 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during PREPARING_REBALANCE state 02.12.2021 17:25:00,918 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011819, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:00,918 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011829, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:00,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 8 ms and fetched 10 records 02.12.2021 17:25:00,919 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:00,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,919 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:00,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 59 iterations. 02.12.2021 17:25:00,923 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119838ms for 120000ms of delay, sleeping for 172ms 02.12.2021 17:25:00,923 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:00,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,935 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,010 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:01,013 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 244 02.12.2021 17:25:01,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,026 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 182 02.12.2021 17:25:01,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,044 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,044 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,062 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59941ms for 60000ms of delay, sleeping for 69ms 02.12.2021 17:25:01,097 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119115ms for 120000ms of delay, sleeping for 895ms 02.12.2021 17:25:01,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 144 02.12.2021 17:25:01,132 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59323ms for 60000ms of delay, sleeping for 687ms 02.12.2021 17:25:01,134 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:01,146 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239963ms for 240000ms of delay, sleeping for 47ms 02.12.2021 17:25:01,195 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239938ms for 240000ms of delay, sleeping for 72ms 02.12.2021 17:25:01,215 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 43 02.12.2021 17:25:01,214 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,222 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,222 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:01,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,265 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,273 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239310ms for 240000ms of delay, sleeping for 700ms 02.12.2021 17:25:01,275 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,280 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:01,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:01,423 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:01,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:01,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,437 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:01,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,527 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:01,544 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:01,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,545 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,545 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:01,628 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:01,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,766 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,769 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,782 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:01,784 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:01,785 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1053) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:01,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:01,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,792 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,793 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 33962ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:01,821 DEBUG [org.apache.kafka.streams.processor.internals.RecordCollectorImpl:247] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Flushing record collector 02.12.2021 17:25:01,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:381] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Prepared RUNNING task for committing 02.12.2021 17:25:01,821 TRACE [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1157] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending OffsetCommit request with {redacted.Cloud.Message.Callback.Retry1-1=OffsetAndMetadata{offset=1155144, leaderEpoch=null, metadata='AQAAAX179bwR'}} to coordinator kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:01,822 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending OFFSET_COMMIT request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1054) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null,topics=[{name=redacted.Cloud.Message.Callback.Retry1,partitions=[{partition_index=1,committed_offset=1155144,committed_leader_epoch=-1,committed_metadata=AQAAAX179bwR}]}]} 02.12.2021 17:25:01,822 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1053): org.apache.kafka.common.requests.HeartbeatResponse@669e9660 02.12.2021 17:25:01,822 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:01,828 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received OFFSET_COMMIT response from node 2147483644 for request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1054): OffsetCommitResponseData(throttleTimeMs=0, topics=[OffsetCommitResponseTopic(name='redacted.Cloud.Message.Callback.Retry1', partitions=[OffsetCommitResponsePartition(partitionIndex=1, errorCode=0)])]) 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1185] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Committed offset 1155144 for partition redacted.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:808] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Checkpointable offsets {redacted.Cloud.Message.Callback.Retry1-1=1155143} 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:465] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Finalized commit for RUNNING task with eos false enforce checkpoint false 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:684] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Committed all active tasks [0_1] and standby tasks [] in 0ms 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 8 iterations. 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 8 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:01,841 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155144, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:01,841 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155154, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:01,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:01,842 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:01,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,842 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:01,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 8 iterations. 02.12.2021 17:25:01,847 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59826ms for 60000ms of delay, sleeping for 184ms 02.12.2021 17:25:01,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:01,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:01,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:01,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,947 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 54 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 153518ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 55 iterations. 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 55 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 153518ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:01,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:01,977 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963426, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:01,977 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963436, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:01,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:01,980 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:01,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,980 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:01,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 55 iterations. 02.12.2021 17:25:01,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239762ms for 240000ms of delay, sleeping for 248ms 02.12.2021 17:25:01,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:01,996 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119158ms for 120000ms of delay, sleeping for 852ms 02.12.2021 17:25:02,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,029 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:02,037 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59025ms for 60000ms of delay, sleeping for 985ms 02.12.2021 17:25:02,039 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:02,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,049 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,049 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:02,140 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:02,193 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 76 02.12.2021 17:25:02,231 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239683ms for 240000ms of delay, sleeping for 327ms 02.12.2021 17:25:02,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:02,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,231 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,232 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 313 02.12.2021 17:25:02,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,271 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:02,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,294 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 212 02.12.2021 17:25:02,371 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:02,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 112 02.12.2021 17:25:02,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,450 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:02,532 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,533 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,533 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,534 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 11 02.12.2021 17:25:02,545 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:02,545 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:02,546 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=742) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:02,545 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:02,546 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=742): org.apache.kafka.common.requests.HeartbeatResponse@720de19c 02.12.2021 17:25:02,546 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:02,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:02,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:02,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,550 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,559 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:02,561 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:02,561 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=671) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:02,561 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239662ms for 240000ms of delay, sleeping for 348ms 02.12.2021 17:25:02,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 211 02.12.2021 17:25:02,561 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=671): org.apache.kafka.common.requests.HeartbeatResponse@5c58c02f 02.12.2021 17:25:02,562 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:02,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 210 02.12.2021 17:25:02,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 209 02.12.2021 17:25:02,647 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:02,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 108 02.12.2021 17:25:02,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,755 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,755 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,764 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,764 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,765 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 7 02.12.2021 17:25:02,764 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,766 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,766 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:02,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,798 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,799 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,858 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119943ms for 120000ms of delay, sleeping for 67ms 02.12.2021 17:25:02,869 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:02,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:02,910 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239221ms for 240000ms of delay, sleeping for 789ms 02.12.2021 17:25:02,931 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119769ms for 120000ms of delay, sleeping for 241ms 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,969 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:02,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:03,025 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59942ms for 60000ms of delay, sleeping for 68ms 02.12.2021 17:25:03,034 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,036 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,038 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,052 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,052 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:03,076 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:03,096 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59901ms for 60000ms of delay, sleeping for 109ms 02.12.2021 17:25:03,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:03,174 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118225ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:03,176 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:03,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 8 records with 8 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 9 iterations. 02.12.2021 17:25:03,208 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59345ms for 60000ms of delay, sleeping for 665ms 02.12.2021 17:25:03,267 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,269 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:03,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,277 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:03,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,307 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:03,379 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,470 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:03,479 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:03,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:03,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,539 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:03,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,559 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:03,580 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:03,671 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:03,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:03,706 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238823ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,771 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:03,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,780 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:03,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,814 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,817 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:03,874 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59771ms for 60000ms of delay, sleeping for 239ms 02.12.2021 17:25:03,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 389 02.12.2021 17:25:03,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 395 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,967 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,967 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:03,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:04,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,040 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,046 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,060 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:04,088 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:04,114 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=734): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:04,114 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:04,114 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:04,114 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:04,114 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:04,114 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=735) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:04,114 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 446 02.12.2021 17:25:04,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 2 records with 9 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,115 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=676): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:04,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 10 iterations. 02.12.2021 17:25:04,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 10 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:04,117 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:04,124 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:04,124 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:04,123 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 1 fetched records at offset FetchPosition{offset=1155154, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:04,124 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155155, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:04,124 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:1199] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Added READ_UNCOMMITTED fetch request for partition redacted.Cloud.Message.Callback.Retry1-1 at position FetchPosition{offset=1155155, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.par.eu.dc.sigfox.com:9093 (id: 2 rack: null)], epoch=absent}} to node kafka-3.data.sta.par.eu.dc.sigfox.com:9093 (id: 2 rack: null) 02.12.2021 17:25:04,125 DEBUG [org.apache.kafka.clients.FetchSessionHandler:259] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Built incremental fetch (sessionId=1077486472, epoch=196) for node 2. Added (), altered (redacted.Cloud.Message.Callback.Retry1-1), removed () out of (OSS.STAGING.Cloud.Message.Callback.Retry1-1) 02.12.2021 17:25:04,125 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:266] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(redacted.Cloud.Message.Callback.Retry1-1), toForget=(), implied=()) to broker kafka-3.data.sta.par.eu.dc.sigfox.com:9093 (id: 2 rack: null) 02.12.2021 17:25:04,125 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending FETCH request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1055) and timeout 30000 to node 2: {replica_id=-1,max_wait_ms=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=1077486472,session_epoch=196,topics=[{topic=redacted.Cloud.Message.Callback.Retry1,partitions=[{partition=1,current_leader_epoch=-1,fetch_offset=1155155,log_start_offset=-1,partition_max_bytes=1048576}]}],forgotten_topics_data=[],rack_id=} 02.12.2021 17:25:04,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 1 records 02.12.2021 17:25:04,124 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:04,129 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=677) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,assignments=[]} 02.12.2021 17:25:04,136 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 1 02.12.2021 17:25:04,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,137 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:04,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 10 iterations. 02.12.2021 17:25:04,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 331 02.12.2021 17:25:04,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 1 records with 10 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:04,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:04,145 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry1-1 because previous request to kafka-3.data.sta.par.eu.dc.sigfox.com:9093 (id: 2 rack: null) has not been processed 02.12.2021 17:25:04,146 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,146 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received FETCH response from node 2 for request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1055): org.apache.kafka.common.requests.FetchResponse@4ee16d30 02.12.2021 17:25:04,146 DEBUG [org.apache.kafka.clients.FetchSessionHandler:463] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Node 2 sent an incremental fetch response with throttleTimeMs = 0 for session 1077486472 with response=(redacted.Cloud.Message.Callback.Retry1-1) 02.12.2021 17:25:04,146 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:315] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Fetch READ_UNCOMMITTED at offset 1155155 for partition redacted.Cloud.Message.Callback.Retry1-1 returned fetch data (error=NONE, highWaterMark=1155337, lastStableOffset = 1155337, logStartOffset = 0, preferredReadReplica = absent, abortedTransactions = null, divergingEpoch =Optional.empty, recordsSizeInBytes=82508) 02.12.2021 17:25:04,146 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1251] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Preparing to read 82508 bytes of data for partition redacted.Cloud.Message.Callback.Retry1-1 with offset FetchPosition{offset=1155155, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.par.eu.dc.sigfox.com:9093 (id: 2 rack: null)], epoch=absent}} 02.12.2021 17:25:04,146 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1275] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating high watermark for partition redacted.Cloud.Message.Callback.Retry1-1 to 1155337 02.12.2021 17:25:04,146 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1280] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating log start offset for partition redacted.Cloud.Message.Callback.Retry1-1 to 0 02.12.2021 17:25:04,146 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1285] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating last stable offset for partition redacted.Cloud.Message.Callback.Retry1-1 to 1155337 02.12.2021 17:25:04,146 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155155, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:04,146 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155165, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:04,147 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:04,147 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:04,147 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,147 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:04,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:04,148 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58899ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:04,179 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119230ms delay out of 120000ms. Remaining 770ms. Executor queue at 0 02.12.2021 17:25:04,180 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119211ms for 120000ms of delay, sleeping for 799ms 02.12.2021 17:25:04,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:04,189 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=677): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:04,189 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:04,189 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:04,189 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 55 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:04,189 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:04,189 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:04,189 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:04,189 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:04,189 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:04,189 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from RUNNING to REBALANCING 02.12.2021 17:25:04,189 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:04,189 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 278 02.12.2021 17:25:04,189 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=735): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:04,188 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:04,191 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:04,191 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 55 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:04,191 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:04,191 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:04,192 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:04,192 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:04,192 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:04,192 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:04,192 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 369 02.12.2021 17:25:04,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:04,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,273 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:04,285 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,285 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,285 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,285 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,285 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,285 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,285 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 177 02.12.2021 17:25:04,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 268 02.12.2021 17:25:04,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,316 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:04,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:04,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:04,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 71 02.12.2021 17:25:04,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 161 02.12.2021 17:25:04,467 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,467 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:04,467 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:04,467 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:04,467 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:04,467 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:04,467 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:04,468 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:04,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:04,500 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 60 02.12.2021 17:25:04,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:04,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,547 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:04,561 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:04,561 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from REBALANCING to RUNNING 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:04,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:04,587 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:04,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:04,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:04,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:04,687 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:04,707 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239824ms delay out of 240000ms. Remaining 176ms. Executor queue at 1 02.12.2021 17:25:04,709 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239815ms for 240000ms of delay, sleeping for 195ms 02.12.2021 17:25:04,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:04,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:04,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:04,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,786 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,790 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,789 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,795 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:04,798 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:04,798 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1056) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:04,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,819 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,870 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:04,875 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:04,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:04,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:04,898 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1056): org.apache.kafka.common.requests.HeartbeatResponse@2e43b868 02.12.2021 17:25:04,898 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:04,907 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239686ms for 240000ms of delay, sleeping for 324ms 02.12.2021 17:25:04,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,970 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,970 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,971 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 91 02.12.2021 17:25:04,981 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119936ms for 120000ms of delay, sleeping for 74ms 02.12.2021 17:25:04,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:04,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:05,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,048 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,048 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 59 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 108390ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 60 iterations. 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 60 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 108390ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:05,061 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011829, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:05,061 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011839, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:05,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,067 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:05,068 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 60 iterations. 02.12.2021 17:25:05,069 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119747ms for 120000ms of delay, sleeping for 263ms 02.12.2021 17:25:05,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,071 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:05,093 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:05,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:05,149 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59900ms delay out of 60000ms. Remaining 100ms. Executor queue at 0 02.12.2021 17:25:05,149 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59936ms for 60000ms of delay, sleeping for 74ms 02.12.2021 17:25:05,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:05,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:05,194 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:05,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:05,244 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58925ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:05,246 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239917ms for 240000ms of delay, sleeping for 93ms 02.12.2021 17:25:05,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:05,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:05,287 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=747): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 18, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 18, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 18, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0])]) 02.12.2021 17:25:05,286 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=686): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:05,291 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,291 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,291 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,288 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:05,287 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 18, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 18, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 18, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 77, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0])]) 02.12.2021 17:25:05,293 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:05,293 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:05,293 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2262f404, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@69a73f60, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@29ef0f98, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@79076d7a, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4cf0748a, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1cfc1e4b, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4dbe4183, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@f9089f3, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@400be8af, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3cf9a8a3, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@37454e6, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@8a08a6a, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@17f14541, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@663c5bac, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@47cef0e9, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4bb1c1a0, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@56775eda, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@52928e83, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@626b7580, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@271214ac, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4b7f1f1a, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@d5f7660, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3421c53e, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3211062, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@309558d3, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@468fc4d7, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7c3de10d} 02.12.2021 17:25:05,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:05,301 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:05,301 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 258 02.12.2021 17:25:05,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 244 02.12.2021 17:25:05,301 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=687) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,assignments=[]} 02.12.2021 17:25:05,296 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 246 02.12.2021 17:25:05,305 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Constructed client metadata {ce52c39d-7e35-47a3-9952-d434faf478ea=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 261b987a-3223-4915-a984-d667c12885af=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7bc41ba9-9d2e-4546-8593-56556fd30ef4=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 33919357-b8b1-49bb-aeb9-83ed9581975c=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 2d8e79dc-f788-4286-9c64-ec6210a1ff21=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 4fd6cb22-f124-4947-a2a1-f45a767078e7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 66529824-42fc-40c4-8d79-aa4feea5a732=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, cce182bf-5033-4aa1-bc49-50235c7a3ef3=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, f055f10f-2377-42c8-a04b-a6080c326d2b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:05,310 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Queueing Call(callName=metadata, deadlineMs=1638462365310, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:05,311 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:05,311 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:05,311 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462365310, tries=0, nextAllowedTryMs=0)] at 1638462305311 02.12.2021 17:25:05,311 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Metadata is ready to use. 02.12.2021 17:25:05,311 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Found least loaded node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) connected with no in-flight requests 02.12.2021 17:25:05,311 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Assigned Call(callName=metadata, deadlineMs=1638462365310, tries=0, nextAllowedTryMs=0) to node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) 02.12.2021 17:25:05,311 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-5.data.sta.redacted:9093 (id: 4 rack: null). correlationId=94 02.12.2021 17:25:05,311 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=94) and timeout 30000 to node 4: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:05,311 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59999) 02.12.2021 17:25:05,312 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:05,312 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462305312 02.12.2021 17:25:05,312 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59998) 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Received METADATA response from node 4 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=94): org.apache.kafka.common.requests.MetadataResponse@50c3a3b9 02.12.2021 17:25:05,312 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:05,312 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Call(callName=metadata, deadlineMs=1638462365310, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:05,312 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462305312 02.12.2021 17:25:05,312 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=182897) 02.12.2021 17:25:05,311 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:05,312 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:05,313 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_1 from staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:05,313 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa due to tasks changing ownership. 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client ce52c39d-7e35-47a3-9952-d434faf478ea to schedule a followup rebalance 02.12.2021 17:25:05,313 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client ce52c39d-7e35-47a3-9952-d434faf478ea per-consumer assignment: 02.12.2021 17:25:05,313 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 261b987a-3223-4915-a984-d667c12885af per-consumer assignment: 02.12.2021 17:25:05,313 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 7bc41ba9-9d2e-4546-8593-56556fd30ef4 per-consumer assignment: 02.12.2021 17:25:05,314 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 33919357-b8b1-49bb-aeb9-83ed9581975c per-consumer assignment: 02.12.2021 17:25:05,314 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_4 from staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:05,316 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 due to tasks changing ownership. 02.12.2021 17:25:05,317 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 to schedule a followup rebalance 02.12.2021 17:25:05,317 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 per-consumer assignment: 02.12.2021 17:25:05,317 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 4fd6cb22-f124-4947-a2a1-f45a767078e7 per-consumer assignment: 02.12.2021 17:25:05,317 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 66529824-42fc-40c4-8d79-aa4feea5a732 per-consumer assignment: 02.12.2021 17:25:05,317 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client cce182bf-5033-4aa1-bc49-50235c7a3ef3 per-consumer assignment: 02.12.2021 17:25:05,317 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client f055f10f-2377-42c8-a04b-a6080c326d2b per-consumer assignment: 02.12.2021 17:25:05,318 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:05,318 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Finished assignment for group at generation 186: {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry2-7], userDataSize=48), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-5], userDataSize=48), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-2], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-3], userDataSize=48), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-0], userDataSize=48), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-6], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-8], userDataSize=48)} 02.12.2021 17:25:05,318 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending leader SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:05,318 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=748) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]}]} 02.12.2021 17:25:05,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:05,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,319 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,333 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118231ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:05,335 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=748): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,335 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=687): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,335 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,335 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:05,335 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,336 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 186 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:05,336 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:05,336 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:05,336 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:05,336 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 186 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:05,336 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:05,336 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:05,336 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:05,336 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:05,336 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:05,336 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:05,337 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from RUNNING to REBALANCING 02.12.2021 17:25:05,337 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:05,337 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:05,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 484 02.12.2021 17:25:05,337 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:05,337 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:05,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 213 02.12.2021 17:25:05,340 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239946ms for 240000ms of delay, sleeping for 64ms 02.12.2021 17:25:05,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:05,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:05,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 157 02.12.2021 17:25:05,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 143 02.12.2021 17:25:05,407 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239459ms for 240000ms of delay, sleeping for 551ms 02.12.2021 17:25:05,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 382 02.12.2021 17:25:05,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 111 02.12.2021 17:25:05,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,471 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,473 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:05,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 43 02.12.2021 17:25:05,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 57 02.12.2021 17:25:05,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 282 02.12.2021 17:25:05,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 10 02.12.2021 17:25:05,545 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,545 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:05,546 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:05,546 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=743) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:05,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 256 02.12.2021 17:25:05,546 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=743): org.apache.kafka.common.requests.HeartbeatResponse@57fa81fe 02.12.2021 17:25:05,546 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:05,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 255 02.12.2021 17:25:05,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 254 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:05,548 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,560 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:05,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:05,560 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=672) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:05,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 241 02.12.2021 17:25:05,560 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=672): org.apache.kafka.common.requests.HeartbeatResponse@eadb96f 02.12.2021 17:25:05,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:05,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 241 02.12.2021 17:25:05,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 241 02.12.2021 17:25:05,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,569 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:05,638 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 181 02.12.2021 17:25:05,648 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 154 02.12.2021 17:25:05,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:05,661 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 140 02.12.2021 17:25:05,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:05,672 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:05,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 81 02.12.2021 17:25:05,749 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 53 02.12.2021 17:25:05,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:05,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 39 02.12.2021 17:25:05,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:05,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:05,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:05,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,803 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,803 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:05,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,806 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,806 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:05,820 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:05,820 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from REBALANCING to RUNNING 02.12.2021 17:25:05,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:05,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,821 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,824 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:05,873 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:05,876 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:05,906 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:05,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:05,925 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:05,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 55 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 157501ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 157501ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:05,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:05,960 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1394): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:05,960 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:05,960 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:05,960 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:05,960 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:05,960 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1395) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,assignments=[]} 02.12.2021 17:25:05,960 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963436, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:05,960 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963446, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:05,960 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1395): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:05,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:05,961 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:05,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,961 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:05,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:05,961 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239912ms for 240000ms of delay, sleeping for 98ms 02.12.2021 17:25:05,972 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,975 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 91 02.12.2021 17:25:05,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,979 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:06,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 282 02.12.2021 17:25:06,008 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:06,060 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=55, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:06,067 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239829ms for 240000ms of delay, sleeping for 181ms 02.12.2021 17:25:06,050 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,068 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,080 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:06,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,080 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:06,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 176 02.12.2021 17:25:06,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:06,180 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:06,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:06,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:06,232 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:06,233 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 70 02.12.2021 17:25:06,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:06,245 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59926ms delay out of 60000ms. Remaining 74ms. Executor queue at 0 02.12.2021 17:25:06,251 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59684ms for 60000ms of delay, sleeping for 326ms 02.12.2021 17:25:06,261 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239265ms for 240000ms of delay, sleeping for 745ms 02.12.2021 17:25:06,280 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:06,281 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:06,281 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:06,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,304 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,306 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:06,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,325 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,333 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119231ms delay out of 120000ms. Remaining 769ms. Executor queue at 0 02.12.2021 17:25:06,334 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119198ms for 120000ms of delay, sleeping for 812ms 02.12.2021 17:25:06,381 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:06,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 191 02.12.2021 17:25:06,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:06,405 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:06,407 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:06,425 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:06,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:06,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:06,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:06,506 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:06,508 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:06,526 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:06,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:06,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,580 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,580 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,580 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:06,582 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59495ms for 60000ms of delay, sleeping for 515ms 02.12.2021 17:25:06,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,582 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,606 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:06,608 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:06,626 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:06,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:06,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:06,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:06,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:06,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:06,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:06,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:06,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:06,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:06,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:06,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,810 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,810 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:06,811 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,812 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,813 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:06,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,828 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,828 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:06,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:06,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:06,910 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:06,913 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:06,929 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,981 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 143 02.12.2021 17:25:06,984 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:06,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:07,007 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239766ms for 240000ms of delay, sleeping for 244ms 02.12.2021 17:25:07,011 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:07,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:07,029 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:07,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 43 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30346ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,083 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 30 02.12.2021 17:25:07,099 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58875ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:07,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:07,114 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:07,114 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 55 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:07,114 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=736) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:07,114 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 470 02.12.2021 17:25:07,114 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=736): org.apache.kafka.common.requests.HeartbeatResponse@19cba80d 02.12.2021 17:25:07,114 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:07,115 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 55 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:07,115 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:07,115 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:07,115 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:07,115 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=737) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:07,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 469 02.12.2021 17:25:07,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:07,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:07,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:07,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:07,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:07,125 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:07,125 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 55 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:07,125 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=678) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:07,126 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 356 02.12.2021 17:25:07,126 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=678): org.apache.kafka.common.requests.HeartbeatResponse@6f60ef8c 02.12.2021 17:25:07,126 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:07,126 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 55 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:07,126 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:07,127 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:07,127 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:07,127 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=679) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:07,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 354 02.12.2021 17:25:07,130 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:07,149 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119298ms for 120000ms of delay, sleeping for 712ms 02.12.2021 17:25:07,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:07,216 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:07,214 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:07,212 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:07,231 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:07,226 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:07,255 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239416ms for 240000ms of delay, sleeping for 594ms 02.12.2021 17:25:07,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:07,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:07,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,325 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 516 ms and fetched 0 records 02.12.2021 17:25:07,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,327 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,327 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:07,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,328 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:07,427 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:07,427 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:07,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:07,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,482 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:07,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:07,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:07,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:07,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,584 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,586 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,586 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,628 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:07,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:07,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:07,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:07,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:07,729 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:07,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:07,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:07,810 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:07,816 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:07,816 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1057) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:07,830 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 458 02.12.2021 17:25:07,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,853 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239679ms for 240000ms of delay, sleeping for 331ms 02.12.2021 17:25:07,862 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119768ms for 120000ms of delay, sleeping for 242ms 02.12.2021 17:25:07,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:07,917 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1057): org.apache.kafka.common.requests.HeartbeatResponse@13450c4f 02.12.2021 17:25:07,917 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:07,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:07,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 356 02.12.2021 17:25:07,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:07,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,983 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,990 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:08,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 276 02.12.2021 17:25:08,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 228 02.12.2021 17:25:08,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 270 02.12.2021 17:25:08,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:08,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,098 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,098 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 203 02.12.2021 17:25:08,099 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59875ms delay out of 60000ms. Remaining 125ms. Executor queue at 0 02.12.2021 17:25:08,096 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,100 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,106 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119882ms for 120000ms of delay, sleeping for 128ms 02.12.2021 17:25:08,111 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59452ms for 60000ms of delay, sleeping for 558ms 02.12.2021 17:25:08,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 127 02.12.2021 17:25:08,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 169 02.12.2021 17:25:08,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 176 02.12.2021 17:25:08,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 102 02.12.2021 17:25:08,203 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239025ms for 240000ms of delay, sleeping for 985ms 02.12.2021 17:25:08,238 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119330ms for 120000ms of delay, sleeping for 680ms 02.12.2021 17:25:08,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 26 02.12.2021 17:25:08,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 69 02.12.2021 17:25:08,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 63 02.12.2021 17:25:08,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:08,303 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,303 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 186 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,303 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=688) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null} 02.12.2021 17:25:08,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 295 02.12.2021 17:25:08,303 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=688): org.apache.kafka.common.requests.HeartbeatResponse@1d3256ef 02.12.2021 17:25:08,303 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:08,304 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 186 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:08,304 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:08,304 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:08,304 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,301 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 186 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,308 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=749) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null} 02.12.2021 17:25:08,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 27 02.12.2021 17:25:08,304 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=689) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:08,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 286 02.12.2021 17:25:08,311 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=749): org.apache.kafka.common.requests.HeartbeatResponse@77944ce8 02.12.2021 17:25:08,316 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:08,316 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 186 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:08,322 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:08,322 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:08,324 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,324 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=750) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:08,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 11 02.12.2021 17:25:08,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:08,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,347 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,341 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 212 02.12.2021 17:25:08,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 517 ms and fetched 0 records 02.12.2021 17:25:08,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,357 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:08,403 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:08,408 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:08,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 108 02.12.2021 17:25:08,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:08,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:08,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,485 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,486 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 7 02.12.2021 17:25:08,551 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,556 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:08,556 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=744) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:08,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 315 02.12.2021 17:25:08,559 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=744): org.apache.kafka.common.requests.HeartbeatResponse@45b8f9b2 02.12.2021 17:25:08,559 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:08,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 307 02.12.2021 17:25:08,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 307 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,563 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:08,563 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=673) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:08,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:08,568 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=673): org.apache.kafka.common.requests.HeartbeatResponse@23ecdadc 02.12.2021 17:25:08,568 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:08,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 280 02.12.2021 17:25:08,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 280 02.12.2021 17:25:08,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 203 02.12.2021 17:25:08,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 174 02.12.2021 17:25:08,678 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59666ms for 60000ms of delay, sleeping for 344ms 02.12.2021 17:25:08,764 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:08,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:08,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,850 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:08,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,852 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:08,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,867 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,932 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119996ms for 120000ms of delay, sleeping for 14ms 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 60 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 112280ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 61 iterations. 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 61 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 112280ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:08,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:08,951 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1140): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:08,951 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:08,951 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:08,951 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:08,951 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:08,951 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1141) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:08,951 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011839, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:08,953 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011849, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:08,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:08,957 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:08,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,957 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:08,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 61 iterations. 02.12.2021 17:25:08,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:08,961 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119828ms for 120000ms of delay, sleeping for 182ms 02.12.2021 17:25:08,968 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:08,979 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,980 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 55 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:08,980 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1396) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null} 02.12.2021 17:25:08,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,987 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,988 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:09,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:09,043 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155165, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:09,044 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155175, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:09,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 6 ms and fetched 10 records 02.12.2021 17:25:09,045 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:09,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,046 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:09,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:09,046 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59991ms for 60000ms of delay, sleeping for 19ms 02.12.2021 17:25:09,053 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1141): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:09,054 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=186, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:09,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:09,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:09,069 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59888ms for 60000ms of delay, sleeping for 122ms 02.12.2021 17:25:09,081 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1396): org.apache.kafka.common.requests.HeartbeatResponse@7800933e 02.12.2021 17:25:09,082 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during COMPLETING_REBALANCE state 02.12.2021 17:25:09,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,103 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,105 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,106 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,146 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119229ms for 120000ms of delay, sleeping for 781ms 02.12.2021 17:25:09,163 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:09,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 160731ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 57 iterations. 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 57 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 160731ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:09,189 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Rebalance failed. 02.12.2021 17:25:09,189 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:09,189 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1397) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:09,190 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963446, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:09,190 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963456, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:09,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:09,190 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:09,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,190 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:09,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 57 iterations. 02.12.2021 17:25:09,190 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 237283ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:09,192 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59745ms for 60000ms of delay, sleeping for 265ms 02.12.2021 17:25:09,263 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:09,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:09,282 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:09,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 522 ms and fetched 0 records 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,374 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,377 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,377 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,377 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,377 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,377 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,377 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,377 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,458 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59751ms for 60000ms of delay, sleeping for 259ms 02.12.2021 17:25:09,477 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:09,478 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:09,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,488 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,489 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:09,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:09,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:09,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:09,726 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59696ms for 60000ms of delay, sleeping for 314ms 02.12.2021 17:25:09,778 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:09,780 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:09,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:09,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,876 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,878 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,878 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,881 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:09,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,885 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,887 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,887 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,887 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,887 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,887 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,887 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,941 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119849ms for 120000ms of delay, sleeping for 161ms 02.12.2021 17:25:09,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:09,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:09,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:09,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,992 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:10,076 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59096ms for 60000ms of delay, sleeping for 914ms 02.12.2021 17:25:10,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,120 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119912ms for 120000ms of delay, sleeping for 98ms 02.12.2021 17:25:10,179 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:10,191 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 238284ms delay out of 240000ms. Remaining 1716ms. Executor queue at 0 02.12.2021 17:25:10,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:10,224 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238317ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:10,236 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119304ms for 120000ms of delay, sleeping for 706ms 02.12.2021 17:25:10,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:10,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:10,377 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,378 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,378 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,390 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,490 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:10,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,493 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,494 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:10,591 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:10,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,609 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,609 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,699 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:10,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:10,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:10,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:10,870 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:10,871 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:10,871 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1058) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:10,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,879 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,880 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,880 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,880 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,880 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,880 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 526 ms and fetched 0 records 02.12.2021 17:25:10,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,916 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,916 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 528 ms and fetched 0 records 02.12.2021 17:25:10,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,922 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,946 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119799ms for 120000ms of delay, sleeping for 211ms 02.12.2021 17:25:10,971 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1058): org.apache.kafka.common.requests.HeartbeatResponse@7213a0ba 02.12.2021 17:25:10,971 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:10,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,994 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,007 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155175, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:11,007 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155185, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:11,007 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:11,013 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59250ms for 60000ms of delay, sleeping for 760ms 02.12.2021 17:25:11,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:11,023 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:11,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,115 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:11,123 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:11,160 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119543ms for 120000ms of delay, sleeping for 467ms 02.12.2021 17:25:11,219 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:11,224 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239317ms delay out of 240000ms. Remaining 683ms. Executor queue at 1 02.12.2021 17:25:11,224 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:11,232 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239303ms for 240000ms of delay, sleeping for 707ms 02.12.2021 17:25:11,330 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:11,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:11,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,381 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,381 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,422 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,422 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 139 02.12.2021 17:25:11,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:11,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,426 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 125 02.12.2021 17:25:11,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,496 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 38 02.12.2021 17:25:11,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 24 02.12.2021 17:25:11,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,555 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:11,555 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:11,555 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=745) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:11,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 371 02.12.2021 17:25:11,557 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=745): org.apache.kafka.common.requests.HeartbeatResponse@1ea6df4e 02.12.2021 17:25:11,557 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:11,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 369 02.12.2021 17:25:11,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 369 02.12.2021 17:25:11,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,561 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:11,561 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:11,561 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=674) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:11,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 361 02.12.2021 17:25:11,562 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=674): org.apache.kafka.common.requests.HeartbeatResponse@49da3339 02.12.2021 17:25:11,562 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:11,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 360 02.12.2021 17:25:11,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 359 02.12.2021 17:25:11,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,616 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:11,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,617 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,617 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,629 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119846ms for 120000ms of delay, sleeping for 164ms 02.12.2021 17:25:11,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 268 02.12.2021 17:25:11,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 259 02.12.2021 17:25:11,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 168 02.12.2021 17:25:11,764 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 158 02.12.2021 17:25:11,774 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59705ms for 60000ms of delay, sleeping for 305ms 02.12.2021 17:25:11,797 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119678ms for 120000ms of delay, sleeping for 332ms 02.12.2021 17:25:11,859 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 67 02.12.2021 17:25:11,865 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 57 02.12.2021 17:25:11,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:11,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,890 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,927 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,928 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:11,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,928 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,931 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:11,985 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238815ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:11,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,998 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,039 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:12,039 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 186 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:12,039 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1142) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null} 02.12.2021 17:25:12,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 391 02.12.2021 17:25:12,029 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:12,109 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59787ms for 60000ms of delay, sleeping for 223ms 02.12.2021 17:25:12,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,124 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,123 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 61 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 115466ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 62 iterations. 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 62 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 115466ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:12,137 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1142): org.apache.kafka.common.requests.HeartbeatResponse@78d2b2e9 02.12.2021 17:25:12,137 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during COMPLETING_REBALANCE state 02.12.2021 17:25:12,137 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Rebalance failed. 02.12.2021 17:25:12,137 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:12,137 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:12,137 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:12,137 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1143) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:12,137 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011849, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:12,138 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011859, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:12,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:12,138 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:12,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,138 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:12,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 62 iterations. 02.12.2021 17:25:12,138 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119993ms for 120000ms of delay, sleeping for 17ms 02.12.2021 17:25:12,140 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:12,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:12,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 275 02.12.2021 17:25:12,163 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119920ms for 120000ms of delay, sleeping for 90ms 02.12.2021 17:25:12,243 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:12,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 169 02.12.2021 17:25:12,267 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119483ms for 120000ms of delay, sleeping for 527ms 02.12.2021 17:25:12,354 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59307ms for 60000ms of delay, sleeping for 703ms 02.12.2021 17:25:12,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 76 02.12.2021 17:25:12,364 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:12,391 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,393 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,393 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:12,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,432 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,432 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:12,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,435 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,439 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,501 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,532 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:12,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:12,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:12,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,626 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,632 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:12,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:12,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,637 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,640 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:12,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:12,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:12,743 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:12,796 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119732ms for 120000ms of delay, sleeping for 278ms 02.12.2021 17:25:12,833 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:12,845 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:12,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,896 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,933 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,934 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:12,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,940 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,986 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239815ms delay out of 240000ms. Remaining 185ms. Executor queue at 0 02.12.2021 17:25:12,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 57 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 164542ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 58 iterations. 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 58 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 164542ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:13,005 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963456, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:13,012 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963466, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:13,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 12 ms and fetched 10 records 02.12.2021 17:25:13,013 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:13,013 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,013 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:13,013 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,013 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 58 iterations. 02.12.2021 17:25:13,015 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238908ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:13,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,024 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,034 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:13,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:13,073 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59273ms for 60000ms of delay, sleeping for 737ms 02.12.2021 17:25:13,082 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119981ms for 120000ms of delay, sleeping for 29ms 02.12.2021 17:25:13,114 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119965ms for 120000ms of delay, sleeping for 45ms 02.12.2021 17:25:13,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:13,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,130 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,130 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:13,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,141 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:13,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,161 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119049ms for 120000ms of delay, sleeping for 961ms 02.12.2021 17:25:13,237 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:13,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:13,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:13,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:13,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:13,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,434 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,434 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,442 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:13,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:13,635 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:13,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,635 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,642 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,643 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,643 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,643 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:13,643 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,644 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,735 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:13,744 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:13,813 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59694ms for 60000ms of delay, sleeping for 316ms 02.12.2021 17:25:13,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:13,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:13,885 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:13,885 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:13,890 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1059) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:13,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,900 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,990 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1059): org.apache.kafka.common.requests.HeartbeatResponse@7979a8a0 02.12.2021 17:25:13,990 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:14,015 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239908ms delay out of 240000ms. Remaining 92ms. Executor queue at 0 02.12.2021 17:25:14,020 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239902ms for 240000ms of delay, sleeping for 108ms 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,036 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:14,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:14,124 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119993ms for 120000ms of delay, sleeping for 17ms 02.12.2021 17:25:14,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:14,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,135 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239757ms for 240000ms of delay, sleeping for 253ms 02.12.2021 17:25:14,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,137 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155185, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:14,137 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155195, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:14,137 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:14,138 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:14,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,137 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,138 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,140 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:14,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:14,153 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119432ms for 120000ms of delay, sleeping for 578ms 02.12.2021 17:25:14,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:14,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,158 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=750): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0])]) 02.12.2021 17:25:14,158 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=689): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:14,161 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:14,161 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:14,161 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:14,161 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:14,162 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=690) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,assignments=[]} 02.12.2021 17:25:14,162 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 477 02.12.2021 17:25:14,161 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 19, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0])]) 02.12.2021 17:25:14,170 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:14,174 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:14,174 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6d5dd109, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@554c1909, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@46eaaf24, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@13cc481d, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@101a06f4, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@25abd1af, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@36abc9b1, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4756f3d9, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7bb6061, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7ce6ef90, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@14e847fc, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6450a683, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@38346a00, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@73e598f9, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@25e4e6de, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@44e1285f, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5f1e3ea9, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6347e3, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2338327, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4f5de0ee, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@611874da, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7b5e6b45, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@656ae5b2, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@45903361, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@670c21ad, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6f006a4a, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5df5556d} 02.12.2021 17:25:14,191 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59535ms for 60000ms of delay, sleeping for 475ms 02.12.2021 17:25:14,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Constructed client metadata {ce52c39d-7e35-47a3-9952-d434faf478ea=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 261b987a-3223-4915-a984-d667c12885af=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7bc41ba9-9d2e-4546-8593-56556fd30ef4=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 33919357-b8b1-49bb-aeb9-83ed9581975c=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 2d8e79dc-f788-4286-9c64-ec6210a1ff21=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 4fd6cb22-f124-4947-a2a1-f45a767078e7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 66529824-42fc-40c4-8d79-aa4feea5a732=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, cce182bf-5033-4aa1-bc49-50235c7a3ef3=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, f055f10f-2377-42c8-a04b-a6080c326d2b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:14,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:14,222 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:14,222 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:14,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:14,222 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Queueing Call(callName=metadata, deadlineMs=1638462374222, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:14,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:14,235 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:14,235 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462374222, tries=0, nextAllowedTryMs=0)] at 1638462314235 02.12.2021 17:25:14,235 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Metadata is ready to use. 02.12.2021 17:25:14,235 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Found least loaded node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) connected with no in-flight requests 02.12.2021 17:25:14,235 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Assigned Call(callName=metadata, deadlineMs=1638462374222, tries=0, nextAllowedTryMs=0) to node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) 02.12.2021 17:25:14,236 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:14,236 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:14,236 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-5.data.sta.redacted:9093 (id: 4 rack: null). correlationId=95 02.12.2021 17:25:14,248 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_1 from staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:14,248 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=95) and timeout 30000 to node 4: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:14,249 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:14,251 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:14,248 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59987) 02.12.2021 17:25:14,248 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa due to tasks changing ownership. 02.12.2021 17:25:14,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client ce52c39d-7e35-47a3-9952-d434faf478ea to schedule a followup rebalance 02.12.2021 17:25:14,260 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client ce52c39d-7e35-47a3-9952-d434faf478ea per-consumer assignment: 02.12.2021 17:25:14,255 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:14,261 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462314261 02.12.2021 17:25:14,260 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 261b987a-3223-4915-a984-d667c12885af per-consumer assignment: 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 7bc41ba9-9d2e-4546-8593-56556fd30ef4 per-consumer assignment: 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 33919357-b8b1-49bb-aeb9-83ed9581975c per-consumer assignment: 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_4 from staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 due to tasks changing ownership. 02.12.2021 17:25:14,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 to schedule a followup rebalance 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 per-consumer assignment: 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 4fd6cb22-f124-4947-a2a1-f45a767078e7 per-consumer assignment: 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 66529824-42fc-40c4-8d79-aa4feea5a732 per-consumer assignment: 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client cce182bf-5033-4aa1-bc49-50235c7a3ef3 per-consumer assignment: 02.12.2021 17:25:14,266 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client f055f10f-2377-42c8-a04b-a6080c326d2b per-consumer assignment: 02.12.2021 17:25:14,267 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:14,267 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Finished assignment for group at generation 187: {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry2-7], userDataSize=48), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-5], userDataSize=48), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-2], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-3], userDataSize=48), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-0], userDataSize=48), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-6], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-8], userDataSize=48)} 02.12.2021 17:25:14,267 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending leader SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:14,267 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=751) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]}]} 02.12.2021 17:25:14,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 133 02.12.2021 17:25:14,265 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59961) 02.12.2021 17:25:14,268 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Received METADATA response from node 4 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=95): org.apache.kafka.common.requests.MetadataResponse@6430b9a9 02.12.2021 17:25:14,268 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:14,268 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Call(callName=metadata, deadlineMs=1638462374222, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:14,268 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462314268 02.12.2021 17:25:14,268 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=173941) 02.12.2021 17:25:14,289 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=751): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,290 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,292 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:14,289 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=690): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,292 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,292 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:14,292 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 187 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:14,292 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:14,293 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,293 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,293 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:14,293 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,293 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from RUNNING to REBALANCING 02.12.2021 17:25:14,293 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:14,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 347 02.12.2021 17:25:14,292 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 187 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:14,296 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:14,300 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,308 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,308 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:14,310 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,316 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:14,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 111 02.12.2021 17:25:14,336 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=679): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:14,336 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:14,338 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:14,336 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=737): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:14,338 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:14,338 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:14,338 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:14,341 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:14,341 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=680) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,assignments=[]} 02.12.2021 17:25:14,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:14,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 186 02.12.2021 17:25:14,341 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:14,350 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:14,350 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=738) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:14,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:14,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:14,379 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=680): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,379 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=738): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,379 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,383 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:14,383 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 56 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:14,383 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:14,384 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,384 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,383 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,385 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:14,385 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:14,387 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,387 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 56 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:14,388 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:14,388 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 244 02.12.2021 17:25:14,388 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,413 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:14,413 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,387 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from RUNNING to REBALANCING 02.12.2021 17:25:14,415 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:14,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 515 ms and fetched 0 records 02.12.2021 17:25:14,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,417 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:14,417 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,417 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,417 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:14,417 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:14,417 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,417 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,415 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 149 02.12.2021 17:25:14,415 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:14,421 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239506ms for 240000ms of delay, sleeping for 504ms 02.12.2021 17:25:14,420 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 267 02.12.2021 17:25:14,417 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:14,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,427 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,437 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 124 02.12.2021 17:25:14,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,447 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,447 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,448 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 107 02.12.2021 17:25:14,512 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 127 02.12.2021 17:25:14,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 8 02.12.2021 17:25:14,522 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 128 02.12.2021 17:25:14,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,529 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:14,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 24 02.12.2021 17:25:14,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 7 02.12.2021 17:25:14,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:14,557 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:14,557 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=746) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:14,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 391 02.12.2021 17:25:14,558 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=746): org.apache.kafka.common.requests.HeartbeatResponse@3badc07 02.12.2021 17:25:14,558 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:14,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 390 02.12.2021 17:25:14,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 390 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,563 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:14,563 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:14,563 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=675) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:14,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 374 02.12.2021 17:25:14,564 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=675): org.apache.kafka.common.requests.HeartbeatResponse@5399db95 02.12.2021 17:25:14,564 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:14,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 373 02.12.2021 17:25:14,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 373 02.12.2021 17:25:14,615 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 24 02.12.2021 17:25:14,622 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 28 02.12.2021 17:25:14,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:14,637 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:14,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:14,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,640 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,641 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,641 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from REBALANCING to RUNNING 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,654 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:14,654 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,654 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,654 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:14,654 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:14,654 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,654 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,654 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,654 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from REBALANCING to RUNNING 02.12.2021 17:25:14,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:14,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,655 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,659 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:14,665 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 272 02.12.2021 17:25:14,669 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59931ms for 60000ms of delay, sleeping for 79ms 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 62 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 118063ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 63 iterations. 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 63 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 118063ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:14,733 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:14,734 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1143): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:14,734 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:14,734 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:14,734 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:14,734 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:14,734 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011859, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:14,734 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011869, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:14,734 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1144) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:14,734 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:14,735 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:14,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,735 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 63 iterations. 02.12.2021 17:25:14,735 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1144): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:14,735 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=187, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:14,735 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119690ms for 120000ms of delay, sleeping for 320ms 02.12.2021 17:25:14,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:14,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:14,760 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:14,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:14,763 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59689ms for 60000ms of delay, sleeping for 321ms 02.12.2021 17:25:14,765 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 172 02.12.2021 17:25:14,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:14,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:14,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:14,862 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:14,862 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 294 02.12.2021 17:25:14,870 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 67 02.12.2021 17:25:14,930 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239953ms for 240000ms of delay, sleeping for 57ms 02.12.2021 17:25:14,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:14,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:14,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,940 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:14,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:14,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,952 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,979 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 176 02.12.2021 17:25:14,995 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239419ms for 240000ms of delay, sleeping for 591ms 02.12.2021 17:25:15,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:15,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,033 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:15,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:15,053 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:15,059 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118983ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:15,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:15,080 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:15,085 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59114ms for 60000ms of delay, sleeping for 896ms 02.12.2021 17:25:15,134 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:15,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:15,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,142 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,143 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:15,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,155 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:15,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:15,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:15,243 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:15,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:15,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:15,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:15,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:15,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:15,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:15,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:15,356 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:15,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:15,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:15,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,442 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,442 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:15,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,453 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,533 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,534 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,534 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,534 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,534 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:15,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:15,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:15,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:15,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:15,592 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239304ms for 240000ms of delay, sleeping for 706ms 02.12.2021 17:25:15,634 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:15,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:15,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,645 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,645 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,655 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:15,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,660 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,660 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,665 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:15,734 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:15,745 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:15,743 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:15,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:15,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:15,758 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:15,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:15,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:15,869 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:15,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 294 02.12.2021 17:25:15,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 75 02.12.2021 17:25:15,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:15,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:15,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:15,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,954 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:15,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,962 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:15,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,978 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 182 02.12.2021 17:25:15,999 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59967ms for 60000ms of delay, sleeping for 43ms 02.12.2021 17:25:16,034 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,047 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,053 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59780ms for 60000ms of delay, sleeping for 230ms 02.12.2021 17:25:16,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:16,060 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119984ms delay out of 120000ms. Remaining 16ms. Executor queue at 0 02.12.2021 17:25:16,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:16,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:16,071 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119897ms for 120000ms of delay, sleeping for 113ms 02.12.2021 17:25:16,078 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:16,080 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 82 02.12.2021 17:25:16,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:16,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,150 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:16,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,172 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:16,179 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:16,198 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119389ms for 120000ms of delay, sleeping for 621ms 02.12.2021 17:25:16,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:16,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:16,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:16,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:16,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:16,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:16,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:16,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:16,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:16,294 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155195, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:16,294 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155205, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:16,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 10 ms and fetched 10 records 02.12.2021 17:25:16,304 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239517ms for 240000ms of delay, sleeping for 493ms 02.12.2021 17:25:16,305 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:16,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,306 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:16,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:16,308 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59600ms for 60000ms of delay, sleeping for 410ms 02.12.2021 17:25:16,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 186 02.12.2021 17:25:16,367 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:16,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 284 02.12.2021 17:25:16,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:16,379 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:16,381 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:16,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 85 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30332ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 183 02.12.2021 17:25:16,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30362ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:16,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,479 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,547 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,551 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,551 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,551 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,551 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,551 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,551 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,552 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 391 02.12.2021 17:25:16,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 391 02.12.2021 17:25:16,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 77 02.12.2021 17:25:16,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:16,579 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:16,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,654 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 291 02.12.2021 17:25:16,673 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:16,679 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:16,734 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59972ms for 60000ms of delay, sleeping for 38ms 02.12.2021 17:25:16,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:16,753 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:16,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:16,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 190 02.12.2021 17:25:16,780 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:16,784 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59736ms for 60000ms of delay, sleeping for 274ms 02.12.2021 17:25:16,799 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239923ms for 240000ms of delay, sleeping for 87ms 02.12.2021 17:25:16,828 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118806ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:16,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 290 02.12.2021 17:25:16,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:16,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:16,879 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:16,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:16,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 84 02.12.2021 17:25:16,892 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239744ms for 240000ms of delay, sleeping for 266ms 02.12.2021 17:25:16,907 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:16,909 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:16,909 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1060) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:16,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,965 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,966 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,969 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,969 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 205 02.12.2021 17:25:16,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:16,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 177 02.12.2021 17:25:16,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 194 02.12.2021 17:25:16,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:16,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,980 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,981 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:17,009 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1060): org.apache.kafka.common.requests.HeartbeatResponse@4acacf1f 02.12.2021 17:25:17,010 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:17,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,053 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,053 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 285 02.12.2021 17:25:17,063 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59972ms for 60000ms of delay, sleeping for 38ms 02.12.2021 17:25:17,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:17,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 104 02.12.2021 17:25:17,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 75 02.12.2021 17:25:17,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:17,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:17,103 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59993ms for 60000ms of delay, sleeping for 17ms 02.12.2021 17:25:17,122 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59046ms for 60000ms of delay, sleeping for 964ms 02.12.2021 17:25:17,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 185 02.12.2021 17:25:17,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,154 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 6 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 58 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 168702ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 59 iterations. 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 59 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 168702ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1397): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:17,160 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:17,160 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:17,161 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1398) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,assignments=[]} 02.12.2021 17:25:17,161 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963466, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:17,161 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963476, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:17,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,161 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,161 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 187 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:17,161 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=691) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null} 02.12.2021 17:25:17,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:17,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 494 02.12.2021 17:25:17,161 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=691): org.apache.kafka.common.requests.HeartbeatResponse@4347aad 02.12.2021 17:25:17,162 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:17,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,161 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:17,162 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 59 iterations. 02.12.2021 17:25:17,162 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 187 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:17,162 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:17,162 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:17,162 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:17,162 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239913ms for 240000ms of delay, sleeping for 97ms 02.12.2021 17:25:17,162 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=692) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:17,163 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 492 02.12.2021 17:25:17,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:17,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 4 02.12.2021 17:25:17,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,175 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 187 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:17,175 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=752) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null} 02.12.2021 17:25:17,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:17,175 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=752): org.apache.kafka.common.requests.HeartbeatResponse@193a7e89 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,176 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:17,176 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 165 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 187 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:17,176 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:17,176 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=753) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:17,176 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 293 02.12.2021 17:25:17,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:17,254 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 84 02.12.2021 17:25:17,261 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:17,261 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1398): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:17,263 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238269ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:17,262 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=56, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:17,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:17,275 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:17,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 65 02.12.2021 17:25:17,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:17,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,340 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,341 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 56 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,341 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=681) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:17,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 213 02.12.2021 17:25:17,341 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=681): org.apache.kafka.common.requests.HeartbeatResponse@2075d91e 02.12.2021 17:25:17,341 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:17,341 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 56 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:17,341 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:17,341 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:17,341 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,342 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=682) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:17,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 211 02.12.2021 17:25:17,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:17,349 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,349 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 56 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,349 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=739) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:17,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 327 02.12.2021 17:25:17,350 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=739): org.apache.kafka.common.requests.HeartbeatResponse@62d08687 02.12.2021 17:25:17,350 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:17,350 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 56 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:17,350 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:17,350 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:17,351 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,351 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=740) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:17,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 325 02.12.2021 17:25:17,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:17,394 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:17,441 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:17,449 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:17,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,466 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:17,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,470 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,470 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,481 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 75 02.12.2021 17:25:17,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,553 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,557 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,557 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:17,557 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=747) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:17,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 425 02.12.2021 17:25:17,557 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=747): org.apache.kafka.common.requests.HeartbeatResponse@4c58ff 02.12.2021 17:25:17,558 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:17,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 424 02.12.2021 17:25:17,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 424 02.12.2021 17:25:17,564 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,565 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:17,565 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=676) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:17,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 402 02.12.2021 17:25:17,565 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=676): org.apache.kafka.common.requests.HeartbeatResponse@309c7648 02.12.2021 17:25:17,566 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:17,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:17,576 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 390 02.12.2021 17:25:17,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,655 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 324 02.12.2021 17:25:17,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,677 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:17,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,678 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,682 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,741 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,750 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 187 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:17,750 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1145) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null} 02.12.2021 17:25:17,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 224 02.12.2021 17:25:17,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:17,828 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119806ms delay out of 120000ms. Remaining 194ms. Executor queue at 0 02.12.2021 17:25:17,831 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119736ms for 120000ms of delay, sleeping for 274ms 02.12.2021 17:25:17,851 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1145): org.apache.kafka.common.requests.HeartbeatResponse@3ee8d591 02.12.2021 17:25:17,851 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during COMPLETING_REBALANCE state 02.12.2021 17:25:17,872 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 111 02.12.2021 17:25:17,881 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:17,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,973 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,973 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:17,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 10 02.12.2021 17:25:17,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,974 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:17,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,984 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,986 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,064 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:18,086 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:18,099 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58203ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:18,110 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 117223ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:18,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:18,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,167 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:18,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,183 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,188 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:18,263 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239269ms delay out of 240000ms. Remaining 731ms. Executor queue at 0 02.12.2021 17:25:18,274 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239168ms for 240000ms of delay, sleeping for 842ms 02.12.2021 17:25:18,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:18,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:18,379 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:18,389 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:18,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:18,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,473 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:18,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,475 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,487 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,487 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,487 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,487 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,487 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,487 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,487 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,487 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,487 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:18,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,567 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:18,588 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:18,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,668 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:18,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,685 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:18,784 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:18,792 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:18,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:18,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:18,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:18,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,982 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,982 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,983 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,984 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,988 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,989 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,069 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:19,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:19,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:19,102 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59206ms delay out of 60000ms. Remaining 794ms. Executor queue at 0 02.12.2021 17:25:19,102 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59334ms for 60000ms of delay, sleeping for 676ms 02.12.2021 17:25:19,110 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 118223ms delay out of 120000ms. Remaining 1777ms. Executor queue at 1 02.12.2021 17:25:19,111 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118089ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:19,117 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239975ms for 240000ms of delay, sleeping for 35ms 02.12.2021 17:25:19,156 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239904ms for 240000ms of delay, sleeping for 106ms 02.12.2021 17:25:19,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:19,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,171 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:19,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:19,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,187 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:19,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:19,285 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238493ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:19,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:19,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:19,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,484 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,489 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,490 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,490 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:19,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,572 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,584 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:19,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:19,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,672 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,672 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:19,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,688 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,689 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:19,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:19,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:19,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:19,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:19,783 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155205, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:19,784 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155215, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:19,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 4 ms and fetched 10 records 02.12.2021 17:25:19,784 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:19,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,784 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:19,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:19,784 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59622ms for 60000ms of delay, sleeping for 388ms 02.12.2021 17:25:19,785 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:19,791 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:19,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:19,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:19,931 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:19,932 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:19,932 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1061) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:19,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:19,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,995 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:20,033 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1061): org.apache.kafka.common.requests.HeartbeatResponse@7156857d 02.12.2021 17:25:20,037 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:20,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:20,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,080 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:20,106 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 394 02.12.2021 17:25:20,111 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119089ms delay out of 120000ms. Remaining 911ms. Executor queue at 1 02.12.2021 17:25:20,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 63 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:20,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 123442ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:20,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:20,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 64 iterations. 02.12.2021 17:25:20,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 64 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:20,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 123443ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:20,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:20,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:20,117 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Rebalance failed. 02.12.2021 17:25:20,117 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:20,117 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:20,117 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:20,117 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1146) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:20,117 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011869, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:20,117 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011879, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:20,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:20,118 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:20,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,118 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:20,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 64 iterations. 02.12.2021 17:25:20,118 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119083ms for 120000ms of delay, sleeping for 927ms 02.12.2021 17:25:20,184 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:20,189 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 56 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:20,189 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1399) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null} 02.12.2021 17:25:20,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,190 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,163 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:20,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,192 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,192 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,191 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59431ms for 60000ms of delay, sleeping for 579ms 02.12.2021 17:25:20,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:20,207 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:20,289 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1399): org.apache.kafka.common.requests.HeartbeatResponse@346a23a4 02.12.2021 17:25:20,287 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239495ms delay out of 240000ms. Remaining 505ms. Executor queue at 2 02.12.2021 17:25:20,292 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239288ms for 240000ms of delay, sleeping for 722ms 02.12.2021 17:25:20,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:20,290 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during COMPLETING_REBALANCE state 02.12.2021 17:25:20,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:20,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:20,422 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 78 02.12.2021 17:25:20,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 69 02.12.2021 17:25:20,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:20,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,500 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 56 02.12.2021 17:25:20,557 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:20,557 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:20,557 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=748) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:20,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 444 02.12.2021 17:25:20,558 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=748): org.apache.kafka.common.requests.HeartbeatResponse@23b442d 02.12.2021 17:25:20,558 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:20,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 443 02.12.2021 17:25:20,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 443 02.12.2021 17:25:20,565 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:20,565 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:20,565 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=677) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:20,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 430 02.12.2021 17:25:20,565 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=677): org.apache.kafka.common.requests.HeartbeatResponse@75e370bd 02.12.2021 17:25:20,565 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:20,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 430 02.12.2021 17:25:20,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 429 02.12.2021 17:25:20,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,582 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,659 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 342 02.12.2021 17:25:20,666 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 329 02.12.2021 17:25:20,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,691 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,693 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,761 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 241 02.12.2021 17:25:20,766 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 229 02.12.2021 17:25:20,785 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59789ms for 60000ms of delay, sleeping for 221ms 02.12.2021 17:25:20,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 140 02.12.2021 17:25:20,866 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 129 02.12.2021 17:25:20,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 39 02.12.2021 17:25:20,967 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 28 02.12.2021 17:25:20,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,996 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,996 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:20,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,999 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,999 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,001 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,008 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59959ms for 60000ms of delay, sleeping for 51ms 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 59 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 172557ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 60 iterations. 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 60 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 172557ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,015 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Rebalance failed. 02.12.2021 17:25:21,015 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:21,015 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:21,016 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:21,016 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1400) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:21,016 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963476, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:21,016 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963486, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:21,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:21,017 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:21,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,017 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 60 iterations. 02.12.2021 17:25:21,017 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238875ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:21,047 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118773ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:21,062 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59180ms for 60000ms of delay, sleeping for 830ms 02.12.2021 17:25:21,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,083 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,098 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:21,099 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,101 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,192 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,192 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,192 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,194 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,194 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:21,202 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:21,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:21,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:21,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:21,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:21,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,497 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:21,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:21,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,499 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,502 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,585 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,600 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:21,602 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,694 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,697 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,697 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,700 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:21,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:21,801 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:21,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:21,902 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:21,903 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59751ms for 60000ms of delay, sleeping for 259ms 02.12.2021 17:25:21,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:21,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:21,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,997 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,002 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:22,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,007 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,019 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,019 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239877ms delay out of 240000ms. Remaining 123ms. Executor queue at 0 02.12.2021 17:25:22,028 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239878ms for 240000ms of delay, sleeping for 132ms 02.12.2021 17:25:22,047 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119773ms delay out of 120000ms. Remaining 227ms. Executor queue at 1 02.12.2021 17:25:22,047 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118991ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:22,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,099 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,103 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:22,121 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:22,164 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59297ms for 60000ms of delay, sleeping for 713ms 02.12.2021 17:25:22,172 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239207ms for 240000ms of delay, sleeping for 803ms 02.12.2021 17:25:22,183 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=740): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:22,183 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:22,183 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:22,183 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:22,183 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,183 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=741) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 13 02.12.2021 17:25:22,183 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=682): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:22,184 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:22,185 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:22,186 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:22,188 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,188 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=683) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,188 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 411 02.12.2021 17:25:22,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,195 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,200 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,202 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,206 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:22,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:22,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:22,317 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=741): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,317 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=683): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,317 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,317 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,318 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:22,318 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:22,319 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 57 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:22,319 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 57 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:22,319 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:22,320 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,319 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:22,321 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,321 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,321 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:22,320 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,323 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:22,323 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,323 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from RUNNING to REBALANCING 02.12.2021 17:25:22,323 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:22,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 282 02.12.2021 17:25:22,323 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,334 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:22,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:22,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 385 02.12.2021 17:25:22,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:22,424 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 175 02.12.2021 17:25:22,436 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:22,441 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 261 02.12.2021 17:25:22,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,498 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:22,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:22,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,509 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:22,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,520 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,524 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:22,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 160 02.12.2021 17:25:22,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:22,605 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,605 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:22,605 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,606 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,606 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:22,606 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:22,606 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,608 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,608 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:22,620 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:22,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 59 02.12.2021 17:25:22,675 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=753): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0])]) 02.12.2021 17:25:22,676 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 79, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0])]) 02.12.2021 17:25:22,676 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:22,676 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:22,676 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1f3d354a, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@73035edd, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@42a33ed, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4e2ecbd7, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@384f203d, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@51d3645f, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@43380d3, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3cfb83f1, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7df4129b, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@37cf60e6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2f624bd4, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4fe6b1c2, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@12db026f, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5e5f8882, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@63de4476, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@194c816e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@53ce2052, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@71f7b931, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@609bae8d, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1d211401, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3f4f7b47, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@733d73ae, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4e6f5482, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5b7bd052, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1faa0d39, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@792ad61, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@33ed3255} 02.12.2021 17:25:22,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,676 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=692): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:22,677 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:22,677 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:22,677 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:22,677 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,677 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=693) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,677 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 18 02.12.2021 17:25:22,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:22,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Constructed client metadata {ce52c39d-7e35-47a3-9952-d434faf478ea=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 261b987a-3223-4915-a984-d667c12885af=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7bc41ba9-9d2e-4546-8593-56556fd30ef4=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 33919357-b8b1-49bb-aeb9-83ed9581975c=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 2d8e79dc-f788-4286-9c64-ec6210a1ff21=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 4fd6cb22-f124-4947-a2a1-f45a767078e7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 66529824-42fc-40c4-8d79-aa4feea5a732=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, cce182bf-5033-4aa1-bc49-50235c7a3ef3=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, f055f10f-2377-42c8-a04b-a6080c326d2b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:22,680 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:22,680 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:22,680 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:22,680 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:22,680 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:22,680 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:22,680 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Queueing Call(callName=metadata, deadlineMs=1638462382680, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:22,680 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:22,681 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462382680, tries=0, nextAllowedTryMs=0)] at 1638462322680 02.12.2021 17:25:22,681 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Metadata is ready to use. 02.12.2021 17:25:22,680 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:22,681 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Found least loaded node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) connected with no in-flight requests 02.12.2021 17:25:22,681 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Assigned Call(callName=metadata, deadlineMs=1638462382680, tries=0, nextAllowedTryMs=0) to node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) 02.12.2021 17:25:22,681 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-5.data.sta.redacted:9093 (id: 4 rack: null). correlationId=96 02.12.2021 17:25:22,681 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=96) and timeout 30000 to node 4: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:22,681 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=60000) 02.12.2021 17:25:22,681 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:22,681 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462322681 02.12.2021 17:25:22,682 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59999) 02.12.2021 17:25:22,682 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Received METADATA response from node 4 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=96): org.apache.kafka.common.requests.MetadataResponse@24ed6b43 02.12.2021 17:25:22,682 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:22,682 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Call(callName=metadata, deadlineMs=1638462382680, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:22,682 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462322682 02.12.2021 17:25:22,682 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=165527) 02.12.2021 17:25:22,681 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:22,684 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:22,684 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_1 from staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:22,684 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa due to tasks changing ownership. 02.12.2021 17:25:22,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client ce52c39d-7e35-47a3-9952-d434faf478ea to schedule a followup rebalance 02.12.2021 17:25:22,684 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client ce52c39d-7e35-47a3-9952-d434faf478ea per-consumer assignment: 02.12.2021 17:25:22,686 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 261b987a-3223-4915-a984-d667c12885af per-consumer assignment: 02.12.2021 17:25:22,692 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 7bc41ba9-9d2e-4546-8593-56556fd30ef4 per-consumer assignment: 02.12.2021 17:25:22,693 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 33919357-b8b1-49bb-aeb9-83ed9581975c per-consumer assignment: 02.12.2021 17:25:22,693 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_4 from staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:22,693 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 due to tasks changing ownership. 02.12.2021 17:25:22,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 to schedule a followup rebalance 02.12.2021 17:25:22,693 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 per-consumer assignment: 02.12.2021 17:25:22,693 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 4fd6cb22-f124-4947-a2a1-f45a767078e7 per-consumer assignment: 02.12.2021 17:25:22,693 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 66529824-42fc-40c4-8d79-aa4feea5a732 per-consumer assignment: 02.12.2021 17:25:22,693 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client cce182bf-5033-4aa1-bc49-50235c7a3ef3 per-consumer assignment: 02.12.2021 17:25:22,694 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client f055f10f-2377-42c8-a04b-a6080c326d2b per-consumer assignment: 02.12.2021 17:25:22,694 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:22,694 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Finished assignment for group at generation 188: {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry2-7], userDataSize=48), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-5], userDataSize=48), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-2], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-3], userDataSize=48), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-0], userDataSize=48), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-6], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-8], userDataSize=48)} 02.12.2021 17:25:22,694 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending leader SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:22,695 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=754) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]}]} 02.12.2021 17:25:22,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,696 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,696 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,696 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,696 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,696 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,696 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,696 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 303 02.12.2021 17:25:22,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:22,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,702 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:22,702 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,702 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,702 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:22,702 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:22,702 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,703 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,703 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from REBALANCING to RUNNING 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:22,710 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:22,713 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=693): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,721 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:22,714 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=754): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,713 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,722 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:22,722 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,729 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:22,729 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 188 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:22,729 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:22,729 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,729 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,729 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:22,729 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,729 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from RUNNING to REBALANCING 02.12.2021 17:25:22,729 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:22,729 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 276 02.12.2021 17:25:22,728 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 188 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:22,730 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:22,736 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,736 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,736 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:22,737 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,737 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:22,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 483 02.12.2021 17:25:22,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:22,809 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:22,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:22,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:22,830 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 168 02.12.2021 17:25:22,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 358 02.12.2021 17:25:22,878 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59108ms for 60000ms of delay, sleeping for 902ms 02.12.2021 17:25:22,905 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:22,910 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:22,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:22,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:22,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 68 02.12.2021 17:25:22,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 257 02.12.2021 17:25:22,949 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:22,951 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:22,951 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1062) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:22,976 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238869ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:22,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:23,002 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:23,002 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:23,002 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:23,002 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:23,002 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:23,002 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:23,002 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:23,003 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:23,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:23,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,003 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,006 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:23,010 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:23,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,012 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:23,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:23,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,023 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,023 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:23,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 156 02.12.2021 17:25:23,048 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119992ms delay out of 120000ms. Remaining 8ms. Executor queue at 0 02.12.2021 17:25:23,049 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119892ms for 120000ms of delay, sleeping for 118ms 02.12.2021 17:25:23,052 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1062): org.apache.kafka.common.requests.HeartbeatResponse@1eafed9d 02.12.2021 17:25:23,052 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:23,103 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:23,106 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:23,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:23,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:23,124 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:23,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 54 02.12.2021 17:25:23,169 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119050ms for 120000ms of delay, sleeping for 960ms 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:23,197 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:23,197 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from REBALANCING to RUNNING 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,197 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,203 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:23,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,204 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,211 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:23,213 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:23,224 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:23,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:23,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:23,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:23,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:23,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:23,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:23,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:23,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:23,407 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:23,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:23,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:23,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:23,500 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:23,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,504 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,507 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:23,512 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:23,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,513 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,513 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 52 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,524 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 33 02.12.2021 17:25:23,557 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:23,557 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:23,557 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=749) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:23,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 467 02.12.2021 17:25:23,558 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=749): org.apache.kafka.common.requests.HeartbeatResponse@2a4fb78b 02.12.2021 17:25:23,558 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:23,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 466 02.12.2021 17:25:23,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 466 02.12.2021 17:25:23,566 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:23,566 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:23,566 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=678) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:23,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 448 02.12.2021 17:25:23,566 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=678): org.apache.kafka.common.requests.HeartbeatResponse@6d38c6c3 02.12.2021 17:25:23,566 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:23,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 447 02.12.2021 17:25:23,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 446 02.12.2021 17:25:23,601 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:23,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:23,608 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:23,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,620 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 346 02.12.2021 17:25:23,660 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 364 02.12.2021 17:25:23,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,698 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,698 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,698 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,704 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:23,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 389 02.12.2021 17:25:23,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 244 02.12.2021 17:25:23,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 256 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:23,781 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155215, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:23,781 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155225, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:23,781 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:23,782 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59779ms for 60000ms of delay, sleeping for 231ms 02.12.2021 17:25:23,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:23,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:23,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 397 02.12.2021 17:25:23,842 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 278 02.12.2021 17:25:23,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 142 02.12.2021 17:25:23,869 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 155 02.12.2021 17:25:23,906 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:23,915 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 290 02.12.2021 17:25:23,914 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 284 02.12.2021 17:25:23,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 174 02.12.2021 17:25:23,979 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 45 02.12.2021 17:25:23,976 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239869ms delay out of 240000ms. Remaining 131ms. Executor queue at 0 02.12.2021 17:25:23,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 42 02.12.2021 17:25:24,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:24,016 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 182 02.12.2021 17:25:24,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 189 02.12.2021 17:25:24,004 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,072 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,072 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,075 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239915ms for 240000ms of delay, sleeping for 95ms 02.12.2021 17:25:24,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 39 02.12.2021 17:25:24,071 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,081 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 593 ms and fetched 0 records 02.12.2021 17:25:24,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,127 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,128 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,132 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119934ms for 120000ms of delay, sleeping for 76ms 02.12.2021 17:25:24,132 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59930ms for 60000ms of delay, sleeping for 80ms 02.12.2021 17:25:24,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,173 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239541ms for 240000ms of delay, sleeping for 469ms 02.12.2021 17:25:24,173 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 25 02.12.2021 17:25:24,173 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 31 02.12.2021 17:25:24,173 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:24,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:24,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,199 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:24,227 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,228 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,229 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59975ms for 60000ms of delay, sleeping for 35ms 02.12.2021 17:25:24,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 372 02.12.2021 17:25:24,258 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:24,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 290 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 64 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 127623ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 65 iterations. 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 65 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 127623ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:24,293 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1146): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:24,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:24,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,294 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:24,329 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:24,329 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:24,329 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:24,330 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 3 fetched records at offset FetchPosition{offset=1011879, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:24,330 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011882, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:24,330 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:1199] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Added READ_UNCOMMITTED fetch request for partition redacted.Cloud.Message.Callback.Retry2-2 at position FetchPosition{offset=1011882, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.par.eu.dc.sigfox.com:9093 (id: 4 rack: null)], epoch=absent}} to node kafka-5.data.sta.par.eu.dc.sigfox.com:9093 (id: 4 rack: null) 02.12.2021 17:25:24,330 DEBUG [org.apache.kafka.clients.FetchSessionHandler:259] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Built incremental fetch (sessionId=474560479, epoch=328) for node 4. Added (), altered (redacted.Cloud.Message.Callback.Retry2-2), removed () out of (OSS.STAGING.Cloud.Message.Callback.Retry2-2) 02.12.2021 17:25:24,330 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:266] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(redacted.Cloud.Message.Callback.Retry2-2), toForget=(), implied=()) to broker kafka-5.data.sta.par.eu.dc.sigfox.com:9093 (id: 4 rack: null) 02.12.2021 17:25:24,330 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1147) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:24,330 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending FETCH request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1148) and timeout 30000 to node 4: {replica_id=-1,max_wait_ms=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=474560479,session_epoch=328,topics=[{topic=redacted.Cloud.Message.Callback.Retry2,partitions=[{partition=2,current_leader_epoch=-1,fetch_offset=1011882,log_start_offset=-1,partition_max_bytes=1048576}]}],forgotten_topics_data=[],rack_id=} 02.12.2021 17:25:24,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 37 ms and fetched 3 records 02.12.2021 17:25:24,331 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 3 02.12.2021 17:25:24,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,331 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:24,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 65 iterations. 02.12.2021 17:25:24,332 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1147): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:24,332 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=188, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:24,329 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59329ms for 60000ms of delay, sleeping for 681ms 02.12.2021 17:25:24,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:24,356 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119576ms for 120000ms of delay, sleeping for 434ms 02.12.2021 17:25:24,360 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 271 02.12.2021 17:25:24,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 283 02.12.2021 17:25:24,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:24,383 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 189 02.12.2021 17:25:24,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 285 02.12.2021 17:25:24,433 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received FETCH response from node 4 for request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1148): org.apache.kafka.common.requests.FetchResponse@66970998 02.12.2021 17:25:24,433 DEBUG [org.apache.kafka.clients.FetchSessionHandler:463] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Node 4 sent an incremental fetch response with throttleTimeMs = 0 for session 474560479 with response=(redacted.Cloud.Message.Callback.Retry2-2) 02.12.2021 17:25:24,433 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:315] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Fetch READ_UNCOMMITTED at offset 1011882 for partition redacted.Cloud.Message.Callback.Retry2-2 returned fetch data (error=NONE, highWaterMark=1012246, lastStableOffset = 1012246, logStartOffset = 0, preferredReadReplica = absent, abortedTransactions = null, divergingEpoch =Optional.empty, recordsSizeInBytes=162413) 02.12.2021 17:25:24,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 293 02.12.2021 17:25:24,461 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 170 02.12.2021 17:25:24,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 180 02.12.2021 17:25:24,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:24,506 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 66 02.12.2021 17:25:24,526 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 173 02.12.2021 17:25:24,540 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:24,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 69 02.12.2021 17:25:24,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:24,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,574 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:24,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,581 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,581 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,626 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 73 02.12.2021 17:25:24,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:24,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,631 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,631 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 87 02.12.2021 17:25:24,644 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239651ms for 240000ms of delay, sleeping for 359ms 02.12.2021 17:25:24,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 529 ms and fetched 0 records 02.12.2021 17:25:24,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,656 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:24,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,701 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,701 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,735 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 396 02.12.2021 17:25:24,729 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,738 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 445 02.12.2021 17:25:24,757 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:24,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:24,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:24,800 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119931ms for 120000ms of delay, sleeping for 79ms 02.12.2021 17:25:24,801 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 345 02.12.2021 17:25:24,837 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:24,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:24,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 3 records with 65 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 128212ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:24,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:24,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 66 iterations. 02.12.2021 17:25:24,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 66 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 128224ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:24,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:24,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:24,894 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Rebalance failed. 02.12.2021 17:25:24,894 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:24,895 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:24,895 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:24,895 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1149) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:24,896 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1251] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Preparing to read 162413 bytes of data for partition redacted.Cloud.Message.Callback.Retry2-2 with offset FetchPosition{offset=1011882, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.par.eu.dc.sigfox.com:9093 (id: 4 rack: null)], epoch=absent}} 02.12.2021 17:25:24,896 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1275] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating high watermark for partition redacted.Cloud.Message.Callback.Retry2-2 to 1012246 02.12.2021 17:25:24,896 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1280] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating log start offset for partition redacted.Cloud.Message.Callback.Retry2-2 to 0 02.12.2021 17:25:24,896 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1285] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating last stable offset for partition redacted.Cloud.Message.Callback.Retry2-2 to 1012246 02.12.2021 17:25:24,896 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011882, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:24,896 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011892, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:24,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:24,896 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:24,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,896 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:24,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 66 iterations. 02.12.2021 17:25:24,897 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118708ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:24,900 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 174 02.12.2021 17:25:24,900 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 181 02.12.2021 17:25:24,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:24,938 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:24,949 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 183 02.12.2021 17:25:24,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 237 02.12.2021 17:25:24,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:25,002 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:25,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 73 02.12.2021 17:25:25,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:25,005 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238817ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:25,035 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59952ms for 60000ms of delay, sleeping for 58ms 02.12.2021 17:25:25,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 79 02.12.2021 17:25:25,054 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 129 02.12.2021 17:25:25,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:25,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,075 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,075 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,083 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 18 iterations. 02.12.2021 17:25:25,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 18 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,098 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155225, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:25,103 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155235, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:25,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 5 ms and fetched 10 records 02.12.2021 17:25:25,104 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:25,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,104 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,104 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 18 iterations. 02.12.2021 17:25:25,104 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59742ms for 60000ms of delay, sleeping for 268ms 02.12.2021 17:25:25,102 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:25,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:25,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,133 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 52 02.12.2021 17:25:25,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 29 02.12.2021 17:25:25,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,157 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,176 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:25,185 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,186 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 57 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:25,186 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=742) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:25,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:25,186 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 52 02.12.2021 17:25:25,186 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=742): org.apache.kafka.common.requests.HeartbeatResponse@1b5abdb4 02.12.2021 17:25:25,186 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:25,186 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 57 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:25,186 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:25,186 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:25,187 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:25,187 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=743) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:25,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 51 02.12.2021 17:25:25,185 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,187 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 57 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:25,187 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=684) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:25,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 448 02.12.2021 17:25:25,187 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=684): org.apache.kafka.common.requests.HeartbeatResponse@1771e20 02.12.2021 17:25:25,188 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:25,188 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 57 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:25,188 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:25,188 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:25,188 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 395 02.12.2021 17:25:25,188 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:25,189 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=685) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:25,189 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 444 02.12.2021 17:25:25,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:25,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,205 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,205 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 472 02.12.2021 17:25:25,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,251 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,258 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:25,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:25,286 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:25,287 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:25,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:25,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 370 02.12.2021 17:25:25,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:25,374 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59430ms for 60000ms of delay, sleeping for 580ms 02.12.2021 17:25:25,381 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:25,389 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:25,407 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 270 02.12.2021 17:25:25,460 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:25,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:25,490 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:25,508 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 169 02.12.2021 17:25:25,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:25,575 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,575 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,575 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,576 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 100 02.12.2021 17:25:25,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,586 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,587 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,608 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 69 02.12.2021 17:25:25,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30256ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:25,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,634 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,658 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,659 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,659 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,659 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,677 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,677 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 188 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,677 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=755) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null} 02.12.2021 17:25:25,677 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:25,677 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=755): org.apache.kafka.common.requests.HeartbeatResponse@36918d3a 02.12.2021 17:25:25,677 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:25,677 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 188 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:25,677 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:25,677 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:25,677 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,677 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=756) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:25,677 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:25,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:25,678 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,678 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 188 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:25,678 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=694) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null} 02.12.2021 17:25:25,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 27 02.12.2021 17:25:25,679 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=694): org.apache.kafka.common.requests.HeartbeatResponse@246f0d3 02.12.2021 17:25:25,679 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:25,679 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 188 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:25,679 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:25,679 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:25,679 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,682 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=695) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:25,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 23 02.12.2021 17:25:25,687 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:25,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:25,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,707 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:25,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:25,764 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,776 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30264ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:25,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:25,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,777 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:25,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,779 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:25,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:25,872 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 287 02.12.2021 17:25:25,891 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:25,897 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119708ms delay out of 120000ms. Remaining 292ms. Executor queue at 0 02.12.2021 17:25:25,915 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119344ms for 120000ms of delay, sleeping for 666ms 02.12.2021 17:25:25,956 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59994ms for 60000ms of delay, sleeping for 16ms 02.12.2021 17:25:25,974 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58879ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:25,974 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,974 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:25,974 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1063) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:25,974 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:25,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:26,009 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239821ms delay out of 240000ms. Remaining 179ms. Executor queue at 1 02.12.2021 17:25:26,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 60 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:26,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 177557ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:26,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:26,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 61 iterations. 02.12.2021 17:25:26,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 61 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:26,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 177563ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:26,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:26,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:26,026 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1400): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:26,026 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:26,026 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:26,029 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:26,029 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:26,030 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963486, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:26,030 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963496, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:26,030 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1401) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,assignments=[]} 02.12.2021 17:25:26,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 6 ms and fetched 10 records 02.12.2021 17:25:26,030 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:26,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,030 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:26,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 61 iterations. 02.12.2021 17:25:26,035 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239128ms for 240000ms of delay, sleeping for 882ms 02.12.2021 17:25:26,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:26,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,079 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,080 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:26,080 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:26,080 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1063): org.apache.kafka.common.requests.HeartbeatResponse@2aa76a40 02.12.2021 17:25:26,085 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:26,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:26,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,095 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,100 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 457 02.12.2021 17:25:26,130 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1401): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:26,130 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=57, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:26,135 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30758ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:26,136 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,161 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,162 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 404 02.12.2021 17:25:26,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 357 02.12.2021 17:25:26,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:26,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,207 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,208 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 304 02.12.2021 17:25:26,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30775ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:26,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 256 02.12.2021 17:25:26,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 204 02.12.2021 17:25:26,403 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 154 02.12.2021 17:25:26,463 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:26,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 54 02.12.2021 17:25:26,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:26,559 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:26,559 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=750) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:26,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 41 02.12.2021 17:25:26,560 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=750): org.apache.kafka.common.requests.HeartbeatResponse@7fc00c21 02.12.2021 17:25:26,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:26,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 40 02.12.2021 17:25:26,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 40 02.12.2021 17:25:26,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 2 02.12.2021 17:25:26,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,566 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=679) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:26,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:26,567 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=679): org.apache.kafka.common.requests.HeartbeatResponse@70e0ed82 02.12.2021 17:25:26,568 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:26,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:26,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:26,579 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,580 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,587 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119803ms for 120000ms of delay, sleeping for 207ms 02.12.2021 17:25:26,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:26,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,614 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:26,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31270ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:26,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:26,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,665 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,665 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:26,715 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:26,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:26,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,719 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,720 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,766 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:26,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,790 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31277ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:26,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:26,790 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,797 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119879ms for 120000ms of delay, sleeping for 131ms 02.12.2021 17:25:26,816 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:26,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:26,916 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:26,919 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239956ms for 240000ms of delay, sleeping for 54ms 02.12.2021 17:25:26,933 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118992ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:26,967 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:26,975 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59880ms delay out of 60000ms. Remaining 120ms. Executor queue at 0 02.12.2021 17:25:26,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59263ms for 60000ms of delay, sleeping for 747ms 02.12.2021 17:25:26,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239364ms for 240000ms of delay, sleeping for 646ms 02.12.2021 17:25:27,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:27,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:27,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,081 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,114 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:27,114 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,114 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,114 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,114 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,114 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,114 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,114 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:27,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,149 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31771ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:27,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:27,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:27,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,165 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,215 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:27,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,220 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:27,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:27,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,299 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31786ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:27,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:27,303 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:27,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:27,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:27,467 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:27,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:27,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:27,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,615 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,635 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238808ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:27,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,651 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32274ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:27,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:27,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,652 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,716 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:27,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:27,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,723 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,728 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,728 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,744 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:27,753 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59904ms for 60000ms of delay, sleeping for 106ms 02.12.2021 17:25:27,768 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:27,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32296ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:27,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,809 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:27,870 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59945ms for 60000ms of delay, sleeping for 65ms 02.12.2021 17:25:27,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:27,923 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:27,933 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119992ms delay out of 120000ms. Remaining 8ms. Executor queue at 0 02.12.2021 17:25:27,937 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119829ms for 120000ms of delay, sleeping for 181ms 02.12.2021 17:25:27,953 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59662ms for 60000ms of delay, sleeping for 348ms 02.12.2021 17:25:27,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 184 02.12.2021 17:25:28,025 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:28,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30360ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,088 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,092 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 75 02.12.2021 17:25:28,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,118 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,126 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119842ms for 120000ms of delay, sleeping for 168ms 02.12.2021 17:25:28,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,154 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32776ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:28,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,171 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,218 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:28,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 515 ms and fetched 0 records 02.12.2021 17:25:28,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,252 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,256 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:28,302 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119998ms for 120000ms of delay, sleeping for 12ms 02.12.2021 17:25:28,310 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,316 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32803ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,318 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119256ms for 120000ms of delay, sleeping for 754ms 02.12.2021 17:25:28,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:28,320 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59957ms for 60000ms of delay, sleeping for 53ms 02.12.2021 17:25:28,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 18 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 19 iterations. 02.12.2021 17:25:28,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 19 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:28,375 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155235, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:28,375 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155245, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:28,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:28,376 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:28,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,376 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:28,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 19 iterations. 02.12.2021 17:25:28,376 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59532ms for 60000ms of delay, sleeping for 478ms 02.12.2021 17:25:28,384 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 287 02.12.2021 17:25:28,420 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:28,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:28,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:28,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:28,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30862ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,619 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,635 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239808ms delay out of 240000ms. Remaining 192ms. Executor queue at 0 02.12.2021 17:25:28,636 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239742ms for 240000ms of delay, sleeping for 268ms 02.12.2021 17:25:28,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,658 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33280ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,672 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:28,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:28,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,757 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,757 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,758 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:28,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,820 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:28,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,821 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,821 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33309ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,822 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,866 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58970ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:28,875 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:28,905 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239992ms for 240000ms of delay, sleeping for 18ms 02.12.2021 17:25:28,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:28,928 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239679ms for 240000ms of delay, sleeping for 331ms 02.12.2021 17:25:28,976 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:28,992 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:28,992 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:28,992 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1064) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:29,022 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:29,040 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:29,040 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 57 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:29,041 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1402) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null} 02.12.2021 17:25:29,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 66 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,075 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 132405ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 67 iterations. 02.12.2021 17:25:29,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 67 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 132406ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:29,076 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011892, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:29,076 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011902, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:29,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:29,077 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:29,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,077 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:29,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 67 iterations. 02.12.2021 17:25:29,077 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119943ms for 120000ms of delay, sleeping for 67ms 02.12.2021 17:25:29,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:29,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31364ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,092 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,092 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1064): org.apache.kafka.common.requests.HeartbeatResponse@3689fa3c 02.12.2021 17:25:29,093 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:29,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,120 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,121 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,121 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 438 02.12.2021 17:25:29,141 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1402): org.apache.kafka.common.requests.HeartbeatResponse@42ca5974 02.12.2021 17:25:29,141 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during COMPLETING_REBALANCE state 02.12.2021 17:25:29,146 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119765ms for 120000ms of delay, sleeping for 245ms 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33781ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,159 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,175 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 391 02.12.2021 17:25:29,223 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 336 02.12.2021 17:25:29,258 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:29,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,259 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,261 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239427ms for 240000ms of delay, sleeping for 583ms 02.12.2021 17:25:29,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33810ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 234 02.12.2021 17:25:29,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:29,393 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119775ms for 120000ms of delay, sleeping for 235ms 02.12.2021 17:25:29,425 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 134 02.12.2021 17:25:29,476 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:29,526 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 33 02.12.2021 17:25:29,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,559 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:29,559 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:29,560 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=751) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:29,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 62 02.12.2021 17:25:29,560 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=751): org.apache.kafka.common.requests.HeartbeatResponse@1d293886 02.12.2021 17:25:29,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:29,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 61 02.12.2021 17:25:29,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 61 02.12.2021 17:25:29,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,569 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:29,571 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:29,571 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=680) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:29,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 108 02.12.2021 17:25:29,572 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=680): org.apache.kafka.common.requests.HeartbeatResponse@231dfbe0 02.12.2021 17:25:29,572 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:29,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:29,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:29,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,593 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31866ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,594 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:29,628 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,628 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,628 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,628 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,628 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,628 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,628 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:29,630 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119740ms for 120000ms of delay, sleeping for 270ms 02.12.2021 17:25:29,660 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,662 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 34284ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 3 02.12.2021 17:25:29,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:29,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,678 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,679 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:29,706 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=695): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:29,706 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:29,706 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:29,706 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:29,706 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:29,707 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=696) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,assignments=[]} 02.12.2021 17:25:29,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 52 02.12.2021 17:25:29,706 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=756): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0])]) 02.12.2021 17:25:29,707 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0])]) 02.12.2021 17:25:29,708 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:29,708 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:29,708 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@791d9d14, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6bf6def3, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@31d32642, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@b11951f, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1ef14c54, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@43ad183f, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6b64d73e, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6cda1e20, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@785d7c03, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@539aefd7, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1e8a1fbd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@229d3952, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3133f466, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@355351c0, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7bead720, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@30f636a6, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1e48741a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@313aeb42, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1cc46992, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1d863faa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@40b360a4, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2635f30b, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@190296a6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@35d58143, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6eb8611d, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@671c6a6a, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6aa67789} 02.12.2021 17:25:29,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Constructed client metadata {ce52c39d-7e35-47a3-9952-d434faf478ea=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 261b987a-3223-4915-a984-d667c12885af=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7bc41ba9-9d2e-4546-8593-56556fd30ef4=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 33919357-b8b1-49bb-aeb9-83ed9581975c=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 2d8e79dc-f788-4286-9c64-ec6210a1ff21=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 4fd6cb22-f124-4947-a2a1-f45a767078e7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 66529824-42fc-40c4-8d79-aa4feea5a732=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, cce182bf-5033-4aa1-bc49-50235c7a3ef3=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, f055f10f-2377-42c8-a04b-a6080c326d2b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:29,710 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Queueing Call(callName=metadata, deadlineMs=1638462389710, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:29,712 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:29,712 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462389710, tries=0, nextAllowedTryMs=0)] at 1638462329712 02.12.2021 17:25:29,712 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Metadata is ready to use. 02.12.2021 17:25:29,712 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Found least loaded node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) connected with no in-flight requests 02.12.2021 17:25:29,712 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Assigned Call(callName=metadata, deadlineMs=1638462389710, tries=0, nextAllowedTryMs=0) to node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) 02.12.2021 17:25:29,712 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-5.data.sta.redacted:9093 (id: 4 rack: null). correlationId=97 02.12.2021 17:25:29,712 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=97) and timeout 30000 to node 4: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:29,712 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59998) 02.12.2021 17:25:29,712 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:29,712 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462329712 02.12.2021 17:25:29,713 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59998) 02.12.2021 17:25:29,712 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:29,714 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:29,714 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:29,713 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Received METADATA response from node 4 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=97): org.apache.kafka.common.requests.MetadataResponse@138008e4 02.12.2021 17:25:29,715 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_1 from staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:29,715 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa due to tasks changing ownership. 02.12.2021 17:25:29,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client ce52c39d-7e35-47a3-9952-d434faf478ea to schedule a followup rebalance 02.12.2021 17:25:29,717 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client ce52c39d-7e35-47a3-9952-d434faf478ea per-consumer assignment: 02.12.2021 17:25:29,717 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 261b987a-3223-4915-a984-d667c12885af per-consumer assignment: 02.12.2021 17:25:29,717 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 7bc41ba9-9d2e-4546-8593-56556fd30ef4 per-consumer assignment: 02.12.2021 17:25:29,717 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 33919357-b8b1-49bb-aeb9-83ed9581975c per-consumer assignment: 02.12.2021 17:25:29,717 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_4 from staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:29,717 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 due to tasks changing ownership. 02.12.2021 17:25:29,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 to schedule a followup rebalance 02.12.2021 17:25:29,717 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 per-consumer assignment: 02.12.2021 17:25:29,715 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:29,717 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Call(callName=metadata, deadlineMs=1638462389710, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:29,717 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 4fd6cb22-f124-4947-a2a1-f45a767078e7 per-consumer assignment: 02.12.2021 17:25:29,725 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 66529824-42fc-40c4-8d79-aa4feea5a732 per-consumer assignment: 02.12.2021 17:25:29,725 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client cce182bf-5033-4aa1-bc49-50235c7a3ef3 per-consumer assignment: 02.12.2021 17:25:29,725 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462329717 02.12.2021 17:25:29,725 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=158492) 02.12.2021 17:25:29,725 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client f055f10f-2377-42c8-a04b-a6080c326d2b per-consumer assignment: 02.12.2021 17:25:29,735 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:29,735 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Finished assignment for group at generation 189: {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry2-7], userDataSize=48), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-5], userDataSize=48), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-2], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-3], userDataSize=48), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-0], userDataSize=48), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-6], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-8], userDataSize=48)} 02.12.2021 17:25:29,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:29,737 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending leader SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:29,742 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=757) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]}]} 02.12.2021 17:25:29,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 344 02.12.2021 17:25:29,766 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=757): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,766 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,766 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:29,767 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 189 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:29,759 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:29,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,767 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,768 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=696): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,768 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,769 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:29,769 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 189 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:29,769 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:29,769 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:29,769 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:29,769 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:29,769 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:29,769 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from RUNNING to REBALANCING 02.12.2021 17:25:29,767 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:29,769 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:29,769 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:29,770 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:29,770 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:29,770 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:29,770 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 328 02.12.2021 17:25:29,769 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:29,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 492 02.12.2021 17:25:29,779 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:29,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,829 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 34316ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,829 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:29,847 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239974ms for 240000ms of delay, sleeping for 36ms 02.12.2021 17:25:29,866 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59970ms delay out of 60000ms. Remaining 30ms. Executor queue at 0 02.12.2021 17:25:29,868 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59966ms for 60000ms of delay, sleeping for 44ms 02.12.2021 17:25:29,870 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 224 02.12.2021 17:25:29,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 391 02.12.2021 17:25:29,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 61 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,884 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:29,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 181436ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 62 iterations. 02.12.2021 17:25:29,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 62 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 181441ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:29,903 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Rebalance failed. 02.12.2021 17:25:29,903 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:29,903 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:29,903 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119996ms for 120000ms of delay, sleeping for 14ms 02.12.2021 17:25:29,903 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:29,909 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1403) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:29,909 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963496, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:29,912 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963506, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:29,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 11 ms and fetched 10 records 02.12.2021 17:25:29,914 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:29,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,914 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:29,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 62 iterations. 02.12.2021 17:25:29,916 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59920ms for 60000ms of delay, sleeping for 90ms 02.12.2021 17:25:29,918 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239971ms for 240000ms of delay, sleeping for 39ms 02.12.2021 17:25:29,933 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119778ms for 120000ms of delay, sleeping for 232ms 02.12.2021 17:25:29,943 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:29,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:29,964 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239503ms for 240000ms of delay, sleeping for 507ms 02.12.2021 17:25:29,971 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 124 02.12.2021 17:25:29,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 291 02.12.2021 17:25:29,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 184 02.12.2021 17:25:30,010 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59276ms for 60000ms of delay, sleeping for 734ms 02.12.2021 17:25:30,044 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:30,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 21 02.12.2021 17:25:30,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 189 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,097 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32369ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,098 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,098 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 71 02.12.2021 17:25:30,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,130 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,132 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 34785ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:30,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,164 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,169 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119625ms for 120000ms of delay, sleeping for 385ms 02.12.2021 17:25:30,180 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:30,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,198 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:30,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:30,268 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,276 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,276 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:30,276 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,280 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,280 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:30,280 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:30,280 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,280 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,280 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,280 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from REBALANCING to RUNNING 02.12.2021 17:25:30,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:30,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,281 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,284 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:30,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 34817ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:30,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:30,387 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:30,387 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:30,405 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:30,407 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=743): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:30,407 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=685): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:30,407 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:30,408 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:30,408 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:30,408 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:30,408 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:30,408 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:30,408 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:30,408 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:30,408 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=686) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,assignments=[]} 02.12.2021 17:25:30,408 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=744) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:30,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 422 02.12.2021 17:25:30,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 256 02.12.2021 17:25:30,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:30,446 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=744): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,446 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=686): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,447 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,447 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,447 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:30,447 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 58 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:30,447 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:30,447 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:30,447 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 58 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:30,448 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:30,448 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:30,448 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:30,448 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:30,448 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:30,448 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:30,448 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from RUNNING to REBALANCING 02.12.2021 17:25:30,448 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:30,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 383 02.12.2021 17:25:30,448 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:30,449 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:30,449 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:30,449 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 217 02.12.2021 17:25:30,473 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239487ms for 240000ms of delay, sleeping for 523ms 02.12.2021 17:25:30,487 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:30,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:30,507 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:30,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:30,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 282 02.12.2021 17:25:30,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 115 02.12.2021 17:25:30,556 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119481ms for 120000ms of delay, sleeping for 529ms 02.12.2021 17:25:30,588 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:30,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:30,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,599 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,605 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,633 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 15 02.12.2021 17:25:30,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 181 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,666 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Committing all active tasks [] and standby tasks [] since 35288ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,666 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,685 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:30,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 392 02.12.2021 17:25:30,748 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59945ms for 60000ms of delay, sleeping for 65ms 02.12.2021 17:25:30,736 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:30,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 79 02.12.2021 17:25:30,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:30,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,785 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,785 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,785 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:30,809 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 290 02.12.2021 17:25:30,818 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59695ms for 60000ms of delay, sleeping for 315ms 02.12.2021 17:25:30,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:30,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,832 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,833 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,833 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from REBALANCING to RUNNING 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Committing all active tasks [] and standby tasks [] since 35322ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:30,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:30,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:30,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:30,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 188 02.12.2021 17:25:30,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:30,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 181 02.12.2021 17:25:30,968 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:30,988 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:30,988 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:30,999 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239826ms for 240000ms of delay, sleeping for 184ms 02.12.2021 17:25:31,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:31,039 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:31,053 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:31,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:31,086 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118893ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:31,088 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:31,089 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:31,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,099 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,099 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,099 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,133 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,134 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,136 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58833ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:31,140 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:31,166 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,167 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,167 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,185 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239238ms for 240000ms of delay, sleeping for 772ms 02.12.2021 17:25:31,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,185 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,186 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:31,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:31,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:31,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:31,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:31,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:31,286 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,291 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,291 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:31,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:31,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,336 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:31,391 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:31,393 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:31,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:31,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:31,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:31,472 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:31,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:31,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:31,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:31,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:31,537 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:31,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:31,592 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:31,594 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:31,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,600 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,600 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,635 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,635 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,637 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:31,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,668 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,668 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,687 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,689 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:31,701 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:31,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:31,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:31,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:31,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:31,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:31,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,794 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,802 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:31,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,837 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,843 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:31,881 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 288 02.12.2021 17:25:31,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:31,906 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 395 02.12.2021 17:25:31,909 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 191 02.12.2021 17:25:31,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:31,955 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 180 02.12.2021 17:25:31,959 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239881ms for 240000ms of delay, sleeping for 129ms 02.12.2021 17:25:31,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 182 02.12.2021 17:25:31,991 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:32,001 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,001 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:32,002 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1065) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:32,006 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 292 02.12.2021 17:25:32,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:32,056 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:32,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:32,087 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 82 02.12.2021 17:25:32,087 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119894ms delay out of 120000ms. Remaining 106ms. Executor queue at 0 02.12.2021 17:25:32,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 67 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:32,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 135418ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:32,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 68 iterations. 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 68 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 135419ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1149): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:32,089 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:32,089 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:32,089 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011902, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:32,089 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011912, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:32,090 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1150) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:32,092 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1150): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:32,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:32,092 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:32,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,092 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:32,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 68 iterations. 02.12.2021 17:25:32,093 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:32,092 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119895ms for 120000ms of delay, sleeping for 115ms 02.12.2021 17:25:32,100 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239107ms for 240000ms of delay, sleeping for 903ms 02.12.2021 17:25:32,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:32,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,102 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,102 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1065): org.apache.kafka.common.requests.HeartbeatResponse@7331ecfd 02.12.2021 17:25:32,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,106 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:32,107 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 187 02.12.2021 17:25:32,136 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59833ms delay out of 60000ms. Remaining 167ms. Executor queue at 1 02.12.2021 17:25:32,137 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59137ms for 60000ms of delay, sleeping for 873ms 02.12.2021 17:25:32,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 417 02.12.2021 17:25:32,159 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 182 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,189 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,189 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 380 02.12.2021 17:25:32,192 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=189, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:32,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:32,211 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:32,220 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119330ms for 120000ms of delay, sleeping for 680ms 02.12.2021 17:25:32,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 317 02.12.2021 17:25:32,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 78 02.12.2021 17:25:32,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:32,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 280 02.12.2021 17:25:32,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:32,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,299 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 407 02.12.2021 17:25:32,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:32,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:32,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,351 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 207 02.12.2021 17:25:32,378 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 294 02.12.2021 17:25:32,390 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:32,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 307 02.12.2021 17:25:32,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:32,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:32,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 106 02.12.2021 17:25:32,478 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 194 02.12.2021 17:25:32,490 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:32,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 207 02.12.2021 17:25:32,513 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:32,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:32,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 6 02.12.2021 17:25:32,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,559 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:32,560 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=752) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:32,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:32,560 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=752): org.apache.kafka.common.requests.HeartbeatResponse@45eb1d22 02.12.2021 17:25:32,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:32,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:32,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 81 02.12.2021 17:25:32,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,570 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,571 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:32,571 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=681) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:32,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 119 02.12.2021 17:25:32,572 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=681): org.apache.kafka.common.requests.HeartbeatResponse@4d1511a6 02.12.2021 17:25:32,572 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:32,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 117 02.12.2021 17:25:32,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 117 02.12.2021 17:25:32,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:32,600 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 106 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 101 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:32,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:32,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 17 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,690 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:32,700 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 6 02.12.2021 17:25:32,706 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,706 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 189 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,706 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=697) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null} 02.12.2021 17:25:32,706 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:32,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:32,707 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=697): org.apache.kafka.common.requests.HeartbeatResponse@1e04f398 02.12.2021 17:25:32,707 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:32,707 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 189 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:32,708 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,708 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 189 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,707 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:32,708 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:32,708 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=758) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null} 02.12.2021 17:25:32,708 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:32,708 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=698) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:32,708 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=758): org.apache.kafka.common.requests.HeartbeatResponse@73603c07 02.12.2021 17:25:32,708 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:32,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:32,708 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 189 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:32,709 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:32,709 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:32,709 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,709 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=759) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:32,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:32,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:32,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:32,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:32,791 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:32,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,800 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,800 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,807 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:32,808 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:32,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:32,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,852 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,852 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,875 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:32,893 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:32,910 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119980ms for 120000ms of delay, sleeping for 30ms 02.12.2021 17:25:32,942 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119644ms for 120000ms of delay, sleeping for 366ms 02.12.2021 17:25:32,945 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:32,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:32,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:32,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:33,005 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239705ms for 240000ms of delay, sleeping for 305ms 02.12.2021 17:25:33,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 19 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 31171ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:33,012 DEBUG [org.apache.kafka.streams.processor.internals.RecordCollectorImpl:247] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Flushing record collector 02.12.2021 17:25:33,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:381] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Prepared RUNNING task for committing 02.12.2021 17:25:33,012 TRACE [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1157] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending OffsetCommit request with {redacted.Cloud.Message.Callback.Retry1-1=OffsetAndMetadata{offset=1155245, leaderEpoch=null, metadata='AQAAAX179jXp'}} to coordinator kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:33,012 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending OFFSET_COMMIT request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1066) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null,topics=[{name=redacted.Cloud.Message.Callback.Retry1,partitions=[{partition_index=1,committed_offset=1155245,committed_leader_epoch=-1,committed_metadata=AQAAAX179jXp}]}]} 02.12.2021 17:25:33,030 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received OFFSET_COMMIT response from node 2147483644 for request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1066): OffsetCommitResponseData(throttleTimeMs=0, topics=[OffsetCommitResponseTopic(name='redacted.Cloud.Message.Callback.Retry1', partitions=[OffsetCommitResponsePartition(partitionIndex=1, errorCode=0)])]) 02.12.2021 17:25:33,031 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1185] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Committed offset 1155245 for partition redacted.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:33,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:808] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Checkpointable offsets {redacted.Cloud.Message.Callback.Retry1-1=1155244} 02.12.2021 17:25:33,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:465] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Finalized commit for RUNNING task with eos false enforce checkpoint false 02.12.2021 17:25:33,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:684] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Committed all active tasks [0_1] and standby tasks [] in 0ms 02.12.2021 17:25:33,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 9 iterations. 02.12.2021 17:25:33,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 9 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:33,032 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155245, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:33,032 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155255, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:33,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:33,042 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:33,042 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,042 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:33,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 9 iterations. 02.12.2021 17:25:33,046 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:33,048 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59902ms for 60000ms of delay, sleeping for 108ms 02.12.2021 17:25:33,055 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:33,076 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:33,093 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:33,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,109 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:33,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:33,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,144 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,144 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,156 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:33,160 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59311ms for 60000ms of delay, sleeping for 699ms 02.12.2021 17:25:33,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,173 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,173 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 235 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,191 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,245 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:33,256 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:33,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 134 02.12.2021 17:25:33,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:33,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,304 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,304 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,309 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119860ms for 120000ms of delay, sleeping for 150ms 02.12.2021 17:25:33,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 62 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 184855ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 63 iterations. 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 63 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 184856ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1403): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:33,314 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:33,314 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:33,315 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:33,315 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1404) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,assignments=[]} 02.12.2021 17:25:33,315 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963506, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:33,316 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963516, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:33,316 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1404): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:33,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:33,316 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:33,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,316 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:33,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,317 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 63 iterations. 02.12.2021 17:25:33,317 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239936ms for 240000ms of delay, sleeping for 74ms 02.12.2021 17:25:33,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:33,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:33,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,355 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 53 02.12.2021 17:25:33,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 34 02.12.2021 17:25:33,392 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239521ms for 240000ms of delay, sleeping for 489ms 02.12.2021 17:25:33,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:33,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,408 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 58 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=745) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:33,408 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:33,409 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 447 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 58 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,409 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=687) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:33,409 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 265 02.12.2021 17:25:33,410 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=687): org.apache.kafka.common.requests.HeartbeatResponse@1009fee0 02.12.2021 17:25:33,410 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:33,410 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 58 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:33,410 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:33,410 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:33,410 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,411 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=688) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:33,411 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 262 02.12.2021 17:25:33,411 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=745): org.apache.kafka.common.requests.HeartbeatResponse@176790ab 02.12.2021 17:25:33,411 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:33,411 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 58 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:33,411 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:33,411 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:33,412 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,412 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=746) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:33,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 443 02.12.2021 17:25:33,416 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=58, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:33,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:33,461 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119804ms for 120000ms of delay, sleeping for 206ms 02.12.2021 17:25:33,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:33,509 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:33,509 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:33,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:33,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:33,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,610 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,653 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,668 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119432ms for 120000ms of delay, sleeping for 578ms 02.12.2021 17:25:33,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:33,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,681 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,681 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,681 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,692 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,692 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:33,792 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:33,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:33,811 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,812 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,816 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,855 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:33,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,865 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,870 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59688ms for 60000ms of delay, sleeping for 322ms 02.12.2021 17:25:33,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,893 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,889 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239986ms for 240000ms of delay, sleeping for 24ms 02.12.2021 17:25:33,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:33,940 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239764ms for 240000ms of delay, sleeping for 246ms 02.12.2021 17:25:33,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:34,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:34,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:34,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,111 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,107 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:34,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:34,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,164 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,164 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,182 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,197 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:34,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,198 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,197 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239992ms for 240000ms of delay, sleeping for 18ms 02.12.2021 17:25:34,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,218 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59150ms for 60000ms of delay, sleeping for 860ms 02.12.2021 17:25:34,222 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239377ms for 240000ms of delay, sleeping for 633ms 02.12.2021 17:25:34,259 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119943ms for 120000ms of delay, sleeping for 67ms 02.12.2021 17:25:34,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:34,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:34,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:34,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,319 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,333 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119892ms for 120000ms of delay, sleeping for 118ms 02.12.2021 17:25:34,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:34,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:34,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 527 ms and fetched 0 records 02.12.2021 17:25:34,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,402 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,403 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 68 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 137782ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:34,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:34,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 69 iterations. 02.12.2021 17:25:34,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 69 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 137782ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:34,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:34,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:34,455 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Rebalance failed. 02.12.2021 17:25:34,456 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:34,456 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:34,456 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:34,456 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1151) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:34,461 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011912, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:34,461 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011922, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:34,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 8 ms and fetched 10 records 02.12.2021 17:25:34,464 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:34,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:34,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 69 iterations. 02.12.2021 17:25:34,467 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:34,468 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119336ms for 120000ms of delay, sleeping for 674ms 02.12.2021 17:25:34,496 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:34,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:34,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:34,602 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:34,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:34,621 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,622 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,622 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,666 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,668 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:34,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,683 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,684 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,703 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:34,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,707 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,715 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:34,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:34,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,822 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,822 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,824 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 391 02.12.2021 17:25:34,883 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239673ms for 240000ms of delay, sleeping for 337ms 02.12.2021 17:25:34,884 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 284 02.12.2021 17:25:34,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,904 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:34,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 181 02.12.2021 17:25:35,013 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:35,013 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:35,013 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1067) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:35,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:35,083 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59956ms for 60000ms of delay, sleeping for 54ms 02.12.2021 17:25:35,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 78 02.12.2021 17:25:35,114 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1067): org.apache.kafka.common.requests.HeartbeatResponse@12133403 02.12.2021 17:25:35,114 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:35,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:35,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,125 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:35,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 9 records with 9 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 10 iterations. 02.12.2021 17:25:35,138 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59096ms for 60000ms of delay, sleeping for 914ms 02.12.2021 17:25:35,145 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119195ms for 120000ms of delay, sleeping for 815ms 02.12.2021 17:25:35,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:35,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,169 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,169 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 390 02.12.2021 17:25:35,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:35,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,185 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:35,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,216 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,216 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 354 02.12.2021 17:25:35,222 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239258ms for 240000ms of delay, sleeping for 752ms 02.12.2021 17:25:35,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:35,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 254 02.12.2021 17:25:35,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,323 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:35,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,405 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,405 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 153 02.12.2021 17:25:35,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:35,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 52 02.12.2021 17:25:35,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,559 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:35,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:35,560 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=753) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:35,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 110 02.12.2021 17:25:35,561 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=753): org.apache.kafka.common.requests.HeartbeatResponse@124f09fb 02.12.2021 17:25:35,561 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:35,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 108 02.12.2021 17:25:35,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 107 02.12.2021 17:25:35,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,571 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:35,571 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:35,571 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=682) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:35,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 145 02.12.2021 17:25:35,572 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=682): org.apache.kafka.common.requests.HeartbeatResponse@6630d819 02.12.2021 17:25:35,572 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:35,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 144 02.12.2021 17:25:35,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 144 02.12.2021 17:25:35,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,626 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,626 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 7 02.12.2021 17:25:35,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,670 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,671 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:35,672 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 44 02.12.2021 17:25:35,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,686 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,686 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,722 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,724 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:35,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:35,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,824 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:35,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:35,906 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:35,906 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,911 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,872 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:35,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 281 02.12.2021 17:25:35,961 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119941ms for 120000ms of delay, sleeping for 69ms 02.12.2021 17:25:35,978 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239794ms for 240000ms of delay, sleeping for 216ms 02.12.2021 17:25:36,020 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 151 02.12.2021 17:25:36,032 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119739ms for 120000ms of delay, sleeping for 271ms 02.12.2021 17:25:36,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 177 02.12.2021 17:25:36,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 1 records with 10 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:36,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:36,060 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155255, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:36,063 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155265, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:36,063 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 8 ms and fetched 10 records 02.12.2021 17:25:36,063 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:36,063 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,063 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:36,063 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,063 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:36,080 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59323ms for 60000ms of delay, sleeping for 687ms 02.12.2021 17:25:36,108 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=698): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:36,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:36,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:36,109 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:36,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:36,110 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=699) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,assignments=[]} 02.12.2021 17:25:36,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 214 02.12.2021 17:25:36,108 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=759): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])]) 02.12.2021 17:25:36,112 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])]) 02.12.2021 17:25:36,117 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:36,117 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:36,117 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5e7d0bff, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5ab66560, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3f54b80, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7107cffd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@471310d1, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2e67e036, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7671731, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@48504807, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@171a4c44, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6ed68c1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@390c29b2, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@73d64ee4, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@36eb44b6, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5b2d99ca, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3347aa4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@42d4a548, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2ef68750, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@19a2a88d, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@39630047, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5efa0cc6, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@44acdc4e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@48c6f296, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6906b2f1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@67941f2b, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@45f8f13b, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3a55e950, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@669855d1} 02.12.2021 17:25:36,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,122 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 49 02.12.2021 17:25:36,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:36,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Constructed client metadata {ce52c39d-7e35-47a3-9952-d434faf478ea=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 261b987a-3223-4915-a984-d667c12885af=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7bc41ba9-9d2e-4546-8593-56556fd30ef4=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 33919357-b8b1-49bb-aeb9-83ed9581975c=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 2d8e79dc-f788-4286-9c64-ec6210a1ff21=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 4fd6cb22-f124-4947-a2a1-f45a767078e7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 66529824-42fc-40c4-8d79-aa4feea5a732=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, cce182bf-5033-4aa1-bc49-50235c7a3ef3=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, f055f10f-2377-42c8-a04b-a6080c326d2b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:36,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:36,156 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:36,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,171 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,166 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:36,174 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:36,182 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:36,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:36,182 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Queueing Call(callName=metadata, deadlineMs=1638462396182, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:36,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:36,182 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:36,182 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462396182, tries=0, nextAllowedTryMs=0)] at 1638462336182 02.12.2021 17:25:36,182 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Metadata is ready to use. 02.12.2021 17:25:36,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,187 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,183 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:36,183 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Found least loaded node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) connected with no in-flight requests 02.12.2021 17:25:36,188 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Assigned Call(callName=metadata, deadlineMs=1638462396182, tries=0, nextAllowedTryMs=0) to node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) 02.12.2021 17:25:36,188 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-5.data.sta.redacted:9093 (id: 4 rack: null). correlationId=98 02.12.2021 17:25:36,188 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=98) and timeout 30000 to node 4: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:36,188 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=60000) 02.12.2021 17:25:36,188 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:36,188 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462336188 02.12.2021 17:25:36,188 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59994) 02.12.2021 17:25:36,189 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Received METADATA response from node 4 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=98): org.apache.kafka.common.requests.MetadataResponse@5e93bf46 02.12.2021 17:25:36,189 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:36,189 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Call(callName=metadata, deadlineMs=1638462396182, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:36,189 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462336189 02.12.2021 17:25:36,189 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=152020) 02.12.2021 17:25:36,188 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:36,191 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_1 from staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:36,191 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa due to tasks changing ownership. 02.12.2021 17:25:36,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client ce52c39d-7e35-47a3-9952-d434faf478ea to schedule a followup rebalance 02.12.2021 17:25:36,193 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client ce52c39d-7e35-47a3-9952-d434faf478ea per-consumer assignment: 02.12.2021 17:25:36,193 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 261b987a-3223-4915-a984-d667c12885af per-consumer assignment: 02.12.2021 17:25:36,193 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 7bc41ba9-9d2e-4546-8593-56556fd30ef4 per-consumer assignment: 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 33919357-b8b1-49bb-aeb9-83ed9581975c per-consumer assignment: 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_4 from staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 due to tasks changing ownership. 02.12.2021 17:25:36,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 to schedule a followup rebalance 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 per-consumer assignment: 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 4fd6cb22-f124-4947-a2a1-f45a767078e7 per-consumer assignment: 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 66529824-42fc-40c4-8d79-aa4feea5a732 per-consumer assignment: 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client cce182bf-5033-4aa1-bc49-50235c7a3ef3 per-consumer assignment: 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client f055f10f-2377-42c8-a04b-a6080c326d2b per-consumer assignment: 02.12.2021 17:25:36,194 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:36,194 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Finished assignment for group at generation 190: {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry2-7], userDataSize=48), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-5], userDataSize=48), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-2], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-3], userDataSize=48), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-0], userDataSize=48), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-6], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-8], userDataSize=48)} 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending leader SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:36,195 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=760) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]}]} 02.12.2021 17:25:36,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:36,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 63 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 187744ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:36,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:36,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 64 iterations. 02.12.2021 17:25:36,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 64 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 187744ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:36,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:36,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:36,203 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Rebalance failed. 02.12.2021 17:25:36,203 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:36,203 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:36,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 574 ms and fetched 0 records 02.12.2021 17:25:36,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,204 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,203 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 39, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:36,205 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1405) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:36,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,207 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963516, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:36,208 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963526, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:36,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 7 ms and fetched 10 records 02.12.2021 17:25:36,210 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:36,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,210 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:36,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 64 iterations. 02.12.2021 17:25:36,223 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:36,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:36,210 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=760): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:36,227 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:36,210 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=699): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:36,227 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:36,227 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:36,227 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 190 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:36,227 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:36,228 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:36,228 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:36,227 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:36,231 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 190 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:36,231 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:36,231 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:36,231 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:36,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,235 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,232 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:36,231 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:36,240 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:36,240 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from RUNNING to REBALANCING 02.12.2021 17:25:36,240 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239976ms for 240000ms of delay, sleeping for 34ms 02.12.2021 17:25:36,240 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:36,240 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:36,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 477 02.12.2021 17:25:36,239 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,240 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:36,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:36,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:36,280 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239426ms for 240000ms of delay, sleeping for 584ms 02.12.2021 17:25:36,318 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119612ms for 120000ms of delay, sleeping for 398ms 02.12.2021 17:25:36,344 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:36,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 363 02.12.2021 17:25:36,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 525 ms and fetched 0 records 02.12.2021 17:25:36,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:36,361 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:36,369 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:36,380 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:36,380 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:36,381 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:36,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:36,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,383 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,394 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 277 02.12.2021 17:25:36,387 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:36,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,430 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,431 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,446 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:36,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 256 02.12.2021 17:25:36,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 174 02.12.2021 17:25:36,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 389 02.12.2021 17:25:36,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:36,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 156 02.12.2021 17:25:36,598 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 289 02.12.2021 17:25:36,598 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 73 02.12.2021 17:25:36,647 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:36,648 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 56 02.12.2021 17:25:36,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:36,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,674 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,690 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,701 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 186 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:36,705 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:36,705 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from REBALANCING to RUNNING 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 69 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 140052ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 70 iterations. 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 70 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 140052ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:36,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:36,724 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1151): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:36,724 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:36,724 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:36,724 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:36,724 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:36,725 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011922, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:36,725 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011932, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:36,725 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1152) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:36,725 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:36,725 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:36,725 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,725 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:36,725 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,725 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 70 iterations. 02.12.2021 17:25:36,726 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119775ms for 120000ms of delay, sleeping for 235ms 02.12.2021 17:25:36,727 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1152): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:36,732 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=190, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:36,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:36,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,740 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,740 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,772 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59933ms for 60000ms of delay, sleeping for 77ms 02.12.2021 17:25:36,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:36,801 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 86 02.12.2021 17:25:36,806 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:36,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:36,857 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58410ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:36,880 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:36,882 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238903ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:36,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:36,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,890 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:36,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,931 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,932 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:36,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:36,963 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119837ms for 120000ms of delay, sleeping for 173ms 02.12.2021 17:25:36,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:36,991 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:37,008 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:37,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:37,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:37,092 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:37,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:37,138 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119292ms for 120000ms of delay, sleeping for 718ms 02.12.2021 17:25:37,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:37,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,175 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,175 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,192 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:37,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:37,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,193 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,193 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,193 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,205 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,205 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:37,243 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,245 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:37,293 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:37,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:37,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:37,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:37,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:37,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,393 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,393 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,393 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,393 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,393 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,393 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30311ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:37,393 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,393 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,406 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:37,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,431 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,431 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:37,476 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:37,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:37,506 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:37,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:37,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:37,594 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:37,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:37,652 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:37,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,678 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,679 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,694 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:37,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,706 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,706 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:37,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,769 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,770 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=746): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:37,770 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=688): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:37,770 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:37,770 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:37,771 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:37,771 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:37,771 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=689) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,assignments=[]} 02.12.2021 17:25:37,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 424 02.12.2021 17:25:37,770 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:37,775 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:37,779 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:37,775 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:37,788 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:37,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:37,791 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=747) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:37,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 124 02.12.2021 17:25:37,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:37,869 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:37,860 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59413ms delay out of 60000ms. Remaining 587ms. Executor queue at 0 02.12.2021 17:25:37,881 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59142ms for 60000ms of delay, sleeping for 868ms 02.12.2021 17:25:37,882 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119914ms for 120000ms of delay, sleeping for 96ms 02.12.2021 17:25:37,882 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239903ms delay out of 240000ms. Remaining 97ms. Executor queue at 1 02.12.2021 17:25:37,884 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239872ms for 240000ms of delay, sleeping for 138ms 02.12.2021 17:25:37,887 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:37,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,893 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,894 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,910 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:37,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,938 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,945 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:37,996 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119892ms for 120000ms of delay, sleeping for 118ms 02.12.2021 17:25:37,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:37,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:38,011 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:38,028 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,052 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:38,052 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1068) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:38,052 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239984ms for 240000ms of delay, sleeping for 26ms 02.12.2021 17:25:38,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:38,083 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239981ms for 240000ms of delay, sleeping for 29ms 02.12.2021 17:25:38,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:38,096 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:38,113 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239159ms for 240000ms of delay, sleeping for 851ms 02.12.2021 17:25:38,114 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:38,116 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119230ms for 120000ms of delay, sleeping for 780ms 02.12.2021 17:25:38,152 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1068): org.apache.kafka.common.requests.HeartbeatResponse@377a96b 02.12.2021 17:25:38,154 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:38,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:38,183 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:38,183 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 372 02.12.2021 17:25:38,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:38,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,199 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:38,200 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,201 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,209 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,212 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,212 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,219 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:38,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,273 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,273 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:38,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 267 02.12.2021 17:25:38,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:38,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:38,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:38,393 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 166 02.12.2021 17:25:38,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:38,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,398 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:38,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 516 ms and fetched 0 records 02.12.2021 17:25:38,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,460 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,463 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:38,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:38,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:38,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 164 02.12.2021 17:25:38,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,559 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:38,560 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=754) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:38,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 127 02.12.2021 17:25:38,563 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=754): org.apache.kafka.common.requests.HeartbeatResponse@6d11ae97 02.12.2021 17:25:38,563 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:38,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 124 02.12.2021 17:25:38,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 124 02.12.2021 17:25:38,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,571 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,572 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:38,572 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=683) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:38,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 202 02.12.2021 17:25:38,572 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=683): org.apache.kafka.common.requests.HeartbeatResponse@11cbb5a0 02.12.2021 17:25:38,572 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:38,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 201 02.12.2021 17:25:38,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:38,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:38,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 64 02.12.2021 17:25:38,664 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 23 02.12.2021 17:25:38,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:38,687 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:38,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,688 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,689 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:38,700 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:38,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,707 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:38,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,720 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,720 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:38,750 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58704ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:38,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:38,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,786 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:38,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:38,800 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:38,820 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:38,887 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:38,891 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,899 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 210 02.12.2021 17:25:38,905 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119866ms for 120000ms of delay, sleeping for 144ms 02.12.2021 17:25:38,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:38,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:38,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,969 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,969 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 64 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 190512ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 65 iterations. 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 65 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 190512ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:38,970 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1405): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:38,971 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:38,971 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:38,971 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:38,971 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:38,971 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1406) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,assignments=[]} 02.12.2021 17:25:38,971 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963526, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:38,975 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963536, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:38,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 5 ms and fetched 10 records 02.12.2021 17:25:38,975 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:38,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,975 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:38,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 65 iterations. 02.12.2021 17:25:38,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238648ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:38,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:38,991 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:39,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 104 02.12.2021 17:25:39,022 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:39,056 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119980ms for 120000ms of delay, sleeping for 30ms 02.12.2021 17:25:39,088 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119930ms for 120000ms of delay, sleeping for 80ms 02.12.2021 17:25:39,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:39,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:39,107 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 2 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,111 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:39,111 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 190 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:39,111 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=700) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null} 02.12.2021 17:25:39,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 288 02.12.2021 17:25:39,111 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=700): org.apache.kafka.common.requests.HeartbeatResponse@110e4b99 02.12.2021 17:25:39,111 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:39,111 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 190 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:39,111 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:39,111 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:39,112 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:39,112 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=701) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:39,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 287 02.12.2021 17:25:39,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:39,119 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:39,119 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 190 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:39,119 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=761) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null} 02.12.2021 17:25:39,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 101 02.12.2021 17:25:39,120 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=761): org.apache.kafka.common.requests.HeartbeatResponse@2132636e 02.12.2021 17:25:39,120 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:39,120 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 190 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:39,120 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:39,120 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:39,121 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:39,121 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=762) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:39,121 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:39,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 70 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 142499ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:39,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:39,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 71 iterations. 02.12.2021 17:25:39,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 71 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 142499ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:39,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:39,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:39,170 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Rebalance failed. 02.12.2021 17:25:39,170 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:39,170 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:39,170 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:39,170 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1153) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:39,170 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011932, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:39,170 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011942, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:39,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:39,171 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:39,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,171 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:39,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 71 iterations. 02.12.2021 17:25:39,171 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119978ms for 120000ms of delay, sleeping for 32ms 02.12.2021 17:25:39,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,190 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:39,206 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119002ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:39,220 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:39,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,222 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,222 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,243 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:39,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,287 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,287 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:39,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:39,391 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:39,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,470 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,470 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:39,491 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:39,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:39,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:39,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:39,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,695 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,695 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:39,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,713 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,716 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,728 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:39,729 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,729 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,729 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,729 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,729 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,729 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,729 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,751 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59705ms delay out of 60000ms. Remaining 295ms. Executor queue at 0 02.12.2021 17:25:39,763 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59927ms for 60000ms of delay, sleeping for 83ms 02.12.2021 17:25:39,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,801 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:39,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,817 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,854 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58862ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,913 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 282 02.12.2021 17:25:39,923 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 395 02.12.2021 17:25:39,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,972 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,973 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,973 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239648ms delay out of 240000ms. Remaining 352ms. Executor queue at 1 02.12.2021 17:25:39,987 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239123ms for 240000ms of delay, sleeping for 887ms 02.12.2021 17:25:40,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 182 02.12.2021 17:25:40,030 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 287 02.12.2021 17:25:40,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:40,134 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 183 02.12.2021 17:25:40,196 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,207 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,207 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,208 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119999ms for 120000ms of delay, sleeping for 11ms 02.12.2021 17:25:40,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:40,227 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:40,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,239 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:40,230 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,240 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119972ms for 120000ms of delay, sleeping for 38ms 02.12.2021 17:25:40,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,313 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119380ms for 120000ms of delay, sleeping for 630ms 02.12.2021 17:25:40,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:40,317 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:40,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,319 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,319 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,402 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,403 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:40,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:40,420 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:40,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,474 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 301 02.12.2021 17:25:40,514 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:40,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:40,615 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:40,621 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,721 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:40,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,743 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,743 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,743 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 28 02.12.2021 17:25:40,764 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 516 ms and fetched 0 records 02.12.2021 17:25:40,765 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,765 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,765 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,765 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,765 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,765 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,765 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,765 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,772 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:40,772 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:40,772 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=690) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:40,772 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:40,772 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 471 02.12.2021 17:25:40,775 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:40,776 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:40,776 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=748) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:40,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:40,808 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:40,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:40,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,823 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,823 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,823 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,854 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59862ms delay out of 60000ms. Remaining 138ms. Executor queue at 0 02.12.2021 17:25:40,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:40,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:40,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:40,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:40,856 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155265, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:40,856 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155275, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:40,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:40,856 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:40,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,857 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:40,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:40,857 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59052ms for 60000ms of delay, sleeping for 958ms 02.12.2021 17:25:40,879 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239979ms for 240000ms of delay, sleeping for 31ms 02.12.2021 17:25:40,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,902 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,902 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,908 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:40,912 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239691ms for 240000ms of delay, sleeping for 319ms 02.12.2021 17:25:40,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:40,947 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119727ms for 120000ms of delay, sleeping for 283ms 02.12.2021 17:25:40,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,980 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:41,029 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:41,061 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,062 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:41,062 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1069) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:41,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:41,130 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:41,162 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1069): org.apache.kafka.common.requests.HeartbeatResponse@7a56fb23 02.12.2021 17:25:41,162 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:41,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:41,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,207 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,207 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,207 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 352 02.12.2021 17:25:41,232 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119034ms for 120000ms of delay, sleeping for 976ms 02.12.2021 17:25:41,232 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:41,236 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239039ms for 240000ms of delay, sleeping for 971ms 02.12.2021 17:25:41,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:41,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,245 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:41,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,266 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 251 02.12.2021 17:25:41,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:41,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,326 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,326 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 245 02.12.2021 17:25:41,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:41,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,403 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 142 02.12.2021 17:25:41,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 145 02.12.2021 17:25:41,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:41,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,498 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=701): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:41,498 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=762): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])]) 02.12.2021 17:25:41,498 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:41,499 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:41,499 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:41,499 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 23, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])]) 02.12.2021 17:25:41,499 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:41,499 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:41,499 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:41,499 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=702) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,assignments=[]} 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@23c13f5a, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@36cb5d1b, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@12d9a0ce, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@204288e8, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@ad35855, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@661cef91, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@399cec4c, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@dad265f, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1dacc64f, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1c4dc20c, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@426da345, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2696ea89, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5d73ceaa, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2fdc3aaa, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@84d39f, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@11d7a3a4, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@521239aa, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1c554028, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3aeb6016, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@168d2e6c, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@55118a7e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6373c4f4, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2f9e35c, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7df84321, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@24e3749a, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@59ca4096, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1464abf0} 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 402 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Constructed client metadata {ce52c39d-7e35-47a3-9952-d434faf478ea=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 261b987a-3223-4915-a984-d667c12885af=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7bc41ba9-9d2e-4546-8593-56556fd30ef4=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 33919357-b8b1-49bb-aeb9-83ed9581975c=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 2d8e79dc-f788-4286-9c64-ec6210a1ff21=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 4fd6cb22-f124-4947-a2a1-f45a767078e7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 66529824-42fc-40c4-8d79-aa4feea5a732=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, cce182bf-5033-4aa1-bc49-50235c7a3ef3=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, f055f10f-2377-42c8-a04b-a6080c326d2b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:41,503 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:41,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:41,504 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:41,504 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:41,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:41,504 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Queueing Call(callName=metadata, deadlineMs=1638462401504, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:41,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:41,504 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:41,505 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462401504, tries=0, nextAllowedTryMs=0)] at 1638462341505 02.12.2021 17:25:41,505 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Metadata is ready to use. 02.12.2021 17:25:41,505 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:41,505 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Found least loaded node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) connected with no in-flight requests 02.12.2021 17:25:41,505 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:41,505 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Assigned Call(callName=metadata, deadlineMs=1638462401504, tries=0, nextAllowedTryMs=0) to node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) 02.12.2021 17:25:41,505 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_1 from staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:41,505 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-5.data.sta.redacted:9093 (id: 4 rack: null). correlationId=99 02.12.2021 17:25:41,505 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa due to tasks changing ownership. 02.12.2021 17:25:41,505 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=99) and timeout 30000 to node 4: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:41,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client ce52c39d-7e35-47a3-9952-d434faf478ea to schedule a followup rebalance 02.12.2021 17:25:41,505 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59999) 02.12.2021 17:25:41,505 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client ce52c39d-7e35-47a3-9952-d434faf478ea per-consumer assignment: 02.12.2021 17:25:41,505 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:41,505 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 261b987a-3223-4915-a984-d667c12885af per-consumer assignment: 02.12.2021 17:25:41,505 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462341505 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 7bc41ba9-9d2e-4546-8593-56556fd30ef4 per-consumer assignment: 02.12.2021 17:25:41,506 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59999) 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 33919357-b8b1-49bb-aeb9-83ed9581975c per-consumer assignment: 02.12.2021 17:25:41,506 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Received METADATA response from node 4 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=99): org.apache.kafka.common.requests.MetadataResponse@3f8fa8cc 02.12.2021 17:25:41,506 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_4 from staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 due to tasks changing ownership. 02.12.2021 17:25:41,506 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Call(callName=metadata, deadlineMs=1638462401504, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:41,506 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462341506 02.12.2021 17:25:41,506 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=146703) 02.12.2021 17:25:41,506 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 to schedule a followup rebalance 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 per-consumer assignment: 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 4fd6cb22-f124-4947-a2a1-f45a767078e7 per-consumer assignment: 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 66529824-42fc-40c4-8d79-aa4feea5a732 per-consumer assignment: 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client cce182bf-5033-4aa1-bc49-50235c7a3ef3 per-consumer assignment: 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client f055f10f-2377-42c8-a04b-a6080c326d2b per-consumer assignment: 02.12.2021 17:25:41,506 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:41,507 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Finished assignment for group at generation 191: {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry2-7], userDataSize=48), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-5], userDataSize=48), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-2], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-3], userDataSize=48), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-0], userDataSize=48), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-6], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-8], userDataSize=48)} 02.12.2021 17:25:41,507 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending leader SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:41,507 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=763) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]}]} 02.12.2021 17:25:41,508 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 259 02.12.2021 17:25:41,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 41 02.12.2021 17:25:41,527 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=702): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,527 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,527 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:41,528 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 191 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:41,528 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:41,528 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,528 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,528 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:41,528 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,528 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from RUNNING to REBALANCING 02.12.2021 17:25:41,528 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:41,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 377 02.12.2021 17:25:41,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 44 02.12.2021 17:25:41,528 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=763): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,529 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,529 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:41,530 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 191 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:41,530 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:41,530 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,530 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,530 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:41,530 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,530 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:41,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 238 02.12.2021 17:25:41,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,559 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,560 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:41,560 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=755) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:41,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 147 02.12.2021 17:25:41,561 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=755): org.apache.kafka.common.requests.HeartbeatResponse@392856bb 02.12.2021 17:25:41,561 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:41,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 146 02.12.2021 17:25:41,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 145 02.12.2021 17:25:41,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,573 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=684) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:41,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 253 02.12.2021 17:25:41,574 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=684): org.apache.kafka.common.requests.HeartbeatResponse@5d459e82 02.12.2021 17:25:41,574 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:41,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 252 02.12.2021 17:25:41,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 252 02.12.2021 17:25:41,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 276 02.12.2021 17:25:41,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 137 02.12.2021 17:25:41,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 45 02.12.2021 17:25:41,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 152 02.12.2021 17:25:41,709 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:41,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,720 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 174 02.12.2021 17:25:41,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 36 02.12.2021 17:25:41,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:41,755 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:41,755 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,755 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,763 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:41,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,767 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:41,767 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,767 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,767 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:41,773 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:41,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 52 02.12.2021 17:25:41,773 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,784 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,784 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:41,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,788 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,788 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,797 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=689): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,797 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=747): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,805 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,805 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=748): org.apache.kafka.common.requests.HeartbeatResponse@215e632b 02.12.2021 17:25:41,805 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:41,806 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:41,806 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:41,806 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:41,806 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,806 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,806 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:41,807 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,805 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,808 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:41,808 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=690): org.apache.kafka.common.requests.HeartbeatResponse@78fd6fe2 02.12.2021 17:25:41,808 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:41,808 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:41,808 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:41,810 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,810 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,811 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:41,811 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,808 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from RUNNING to REBALANCING 02.12.2021 17:25:41,811 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:41,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 176 02.12.2021 17:25:41,811 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:41,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 458 02.12.2021 17:25:41,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 451 02.12.2021 17:25:41,817 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59987ms for 60000ms of delay, sleeping for 23ms 02.12.2021 17:25:41,830 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 74 02.12.2021 17:25:41,834 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 391 02.12.2021 17:25:41,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:41,837 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,841 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,849 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:41,849 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59768ms for 60000ms of delay, sleeping for 242ms 02.12.2021 17:25:41,883 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1406): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,888 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:41,888 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,891 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:41,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,911 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,911 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from REBALANCING to RUNNING 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 70 02.12.2021 17:25:41,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 351 02.12.2021 17:25:41,934 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:41,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,984 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,984 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,991 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:41,992 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,992 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:41,992 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1407) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null} 02.12.2021 17:25:42,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 249 02.12.2021 17:25:42,011 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:42,046 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:42,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:42,085 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,093 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59908ms for 60000ms of delay, sleeping for 102ms 02.12.2021 17:25:42,094 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1407): org.apache.kafka.common.requests.HeartbeatResponse@3b390ca4 02.12.2021 17:25:42,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:42,094 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:42,114 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 149 02.12.2021 17:25:42,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:42,146 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:42,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:42,186 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:42,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:42,203 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59651ms for 60000ms of delay, sleeping for 359ms 02.12.2021 17:25:42,212 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119066ms for 120000ms of delay, sleeping for 944ms 02.12.2021 17:25:42,217 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 47 02.12.2021 17:25:42,210 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239338ms for 240000ms of delay, sleeping for 672ms 02.12.2021 17:25:42,220 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 191 02.12.2021 17:25:42,227 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:42,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,229 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,229 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,231 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:42,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:42,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:42,266 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:42,266 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:42,267 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:42,267 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:42,267 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:42,267 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:42,268 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:42,268 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:42,268 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from REBALANCING to RUNNING 02.12.2021 17:25:42,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:42,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,272 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:42,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:42,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,290 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,291 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:42,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:42,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,350 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,354 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,377 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 396 02.12.2021 17:25:42,387 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:42,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 394 02.12.2021 17:25:42,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:42,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,415 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,416 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:42,455 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:42,477 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 295 02.12.2021 17:25:42,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:42,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,491 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:42,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:42,534 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:42,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:42,566 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59588ms for 60000ms of delay, sleeping for 422ms 02.12.2021 17:25:42,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:42,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:42,597 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:42,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:42,634 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:42,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:42,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:42,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:42,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:42,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:42,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:42,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,741 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,766 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:42,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:42,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,773 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,792 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,792 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:42,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:42,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:42,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:42,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,859 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,859 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,873 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,892 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239995ms for 240000ms of delay, sleeping for 15ms 02.12.2021 17:25:42,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:42,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:42,909 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239701ms for 240000ms of delay, sleeping for 309ms 02.12.2021 17:25:42,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,917 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,917 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:42,959 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:42,973 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:42,992 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59269ms for 60000ms of delay, sleeping for 741ms 02.12.2021 17:25:42,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:42,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,993 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,993 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:43,018 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:43,044 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:43,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:43,074 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:43,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:43,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:43,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:43,145 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:43,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:43,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 71 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,168 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 146498ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:43,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:43,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 72 iterations. 02.12.2021 17:25:43,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 72 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 146499ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:43,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:43,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:43,170 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1153): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:43,170 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:43,170 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:43,170 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:43,170 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:43,170 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011942, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:43,170 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011952, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:43,170 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1154) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:43,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:43,171 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:43,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,171 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:43,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,171 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 72 iterations. 02.12.2021 17:25:43,173 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119971ms for 120000ms of delay, sleeping for 39ms 02.12.2021 17:25:43,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:43,196 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:43,196 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:43,219 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:43,221 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239984ms for 240000ms of delay, sleeping for 26ms 02.12.2021 17:25:43,223 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119767ms for 120000ms of delay, sleeping for 243ms 02.12.2021 17:25:43,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,246 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,249 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,251 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 65 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 194794ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:43,252 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:43,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 66 iterations. 02.12.2021 17:25:43,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 66 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 194795ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:43,254 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:43,254 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:43,256 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 02.12.2021 17:25:43,256 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:43,256 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-2], userDataSize=48) 02.12.2021 17:25:43,257 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:43,257 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:330] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Skip resuming since state is RUNNING 02.12.2021 17:25:43,257 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:43,257 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:43,257 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from RUNNING to REBALANCING 02.12.2021 17:25:43,257 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [redacted.Cloud.Message.Callback.Retry3-2] 02.12.2021 17:25:43,257 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,257 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:43,271 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1154): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:43,271 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=191, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:43,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:43,274 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,283 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 489 02.12.2021 17:25:43,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,293 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,293 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:43,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:43,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:43,358 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,358 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:43,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:43,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,384 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 388 02.12.2021 17:25:43,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:43,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:43,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,418 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,450 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:43,458 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,458 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,458 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:43,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:43,469 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119343ms for 120000ms of delay, sleeping for 667ms 02.12.2021 17:25:43,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 288 02.12.2021 17:25:43,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,496 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 279 02.12.2021 17:25:43,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:43,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:43,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:43,559 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,559 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:43,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:43,584 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:43,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 179 02.12.2021 17:25:43,598 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:43,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:43,651 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:43,659 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,659 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,659 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:43,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:43,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 87 02.12.2021 17:25:43,698 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 77 02.12.2021 17:25:43,700 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:43,722 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:43,741 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59635ms for 60000ms of delay, sleeping for 375ms 02.12.2021 17:25:43,757 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:43,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:43,759 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:43,759 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:43,759 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:43,759 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:43,759 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [redacted.Cloud.Message.Callback.Retry3-2] 02.12.2021 17:25:43,759 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:43,759 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:43,760 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:43,760 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:43,760 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from REBALANCING to RUNNING 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 66 iterations. 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 66 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 195303ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.RecordCollectorImpl:247] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Flushing record collector 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:381] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Prepared RUNNING task for committing 02.12.2021 17:25:43,761 TRACE [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1157] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending OffsetCommit request with {redacted.Cloud.Message.Callback.Retry3-2=OffsetAndMetadata{offset=963536, leaderEpoch=null, metadata='AQAAAX17857F'}} to coordinator kafka-5.data.sta.par.eu.dc.sigfox.com:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,761 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending OFFSET_COMMIT request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1408) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,topics=[{name=redacted.Cloud.Message.Callback.Retry3,partitions=[{partition_index=2,committed_offset=963536,committed_leader_epoch=-1,committed_metadata=AQAAAX17857F}]}]} 02.12.2021 17:25:43,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:43,763 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,764 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,766 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:43,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,775 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:43,778 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,778 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=749) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:43,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,779 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=749): org.apache.kafka.common.requests.HeartbeatResponse@53a8d98a 02.12.2021 17:25:43,779 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:43,775 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:43,780 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,766 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,780 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=691) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,784 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:43,779 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:43,785 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:43,786 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:43,785 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=691): org.apache.kafka.common.requests.HeartbeatResponse@764ee4ca 02.12.2021 17:25:43,786 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:43,786 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:43,786 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:43,786 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:43,786 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,787 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=692) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:43,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 497 02.12.2021 17:25:43,784 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received OFFSET_COMMIT response from node 2147483643 for request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1408): OffsetCommitResponseData(throttleTimeMs=0, topics=[OffsetCommitResponseTopic(name='redacted.Cloud.Message.Callback.Retry3', partitions=[OffsetCommitResponsePartition(partitionIndex=2, errorCode=0)])]) 02.12.2021 17:25:43,781 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,788 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,787 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1185] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Committed offset 963536 for partition redacted.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:43,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:808] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Checkpointable offsets {redacted.Cloud.Message.Callback.Retry3-2=963535} 02.12.2021 17:25:43,786 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,791 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=750) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:43,791 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 205 02.12.2021 17:25:43,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:465] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Finalized commit for RUNNING task with eos false enforce checkpoint false 02.12.2021 17:25:43,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:684] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Committed all active tasks [0_2] and standby tasks [] in 0ms 02.12.2021 17:25:43,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:43,796 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963536, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:43,796 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963546, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,798 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:43,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,798 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 66 iterations. 02.12.2021 17:25:43,805 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239804ms for 240000ms of delay, sleeping for 206ms 02.12.2021 17:25:43,823 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:43,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,873 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,873 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,873 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,879 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:43,889 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:43,885 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:43,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:43,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,918 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,973 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:43,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:43,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:43,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:44,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,000 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,014 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239700ms for 240000ms of delay, sleeping for 310ms 02.12.2021 17:25:44,020 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:44,068 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,068 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:44,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:44,073 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1070) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:44,089 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:44,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:44,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,121 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:44,126 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:44,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,133 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:44,136 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155275, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:44,136 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155285, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:44,136 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1070): org.apache.kafka.common.requests.HeartbeatResponse@6b91c7e6 02.12.2021 17:25:44,138 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 5 ms and fetched 10 records 02.12.2021 17:25:44,141 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119967ms for 120000ms of delay, sleeping for 43ms 02.12.2021 17:25:44,142 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:44,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,142 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:44,144 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59920ms for 60000ms of delay, sleeping for 90ms 02.12.2021 17:25:44,186 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119575ms for 120000ms of delay, sleeping for 435ms 02.12.2021 17:25:44,186 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:44,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:44,191 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:44,206 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 90 02.12.2021 17:25:44,285 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,285 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:44,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:44,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,290 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 269 02.12.2021 17:25:44,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:44,285 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,292 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:44,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,299 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,299 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 200 02.12.2021 17:25:44,329 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239991ms for 240000ms of delay, sleeping for 19ms 02.12.2021 17:25:44,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:44,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,385 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59681ms for 60000ms of delay, sleeping for 329ms 02.12.2021 17:25:44,385 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239419ms for 240000ms of delay, sleeping for 591ms 02.12.2021 17:25:44,391 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 168 02.12.2021 17:25:44,383 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,391 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,391 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,391 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,391 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 30 02.12.2021 17:25:44,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,403 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 170 02.12.2021 17:25:44,424 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 75 02.12.2021 17:25:44,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:44,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,434 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 61 02.12.2021 17:25:44,491 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 68 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,502 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,502 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 191 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,502 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,502 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 191 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,502 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=703) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null} 02.12.2021 17:25:44,503 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 436 02.12.2021 17:25:44,502 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=764) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null} 02.12.2021 17:25:44,503 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=703): org.apache.kafka.common.requests.HeartbeatResponse@36c8ba07 02.12.2021 17:25:44,504 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:44,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 295 02.12.2021 17:25:44,504 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=764): org.apache.kafka.common.requests.HeartbeatResponse@2b3a0316 02.12.2021 17:25:44,504 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:44,504 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 191 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:44,504 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 191 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:44,505 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:44,505 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:44,505 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,505 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=765) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:44,505 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:44,505 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:44,506 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:44,506 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,507 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=704) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:44,510 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 63 02.12.2021 17:25:44,507 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 431 02.12.2021 17:25:44,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,562 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,562 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:44,562 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=756) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:44,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 228 02.12.2021 17:25:44,563 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=756): org.apache.kafka.common.requests.HeartbeatResponse@24375eb1 02.12.2021 17:25:44,563 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:44,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 227 02.12.2021 17:25:44,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 227 02.12.2021 17:25:44,573 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:44,574 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:44,574 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=685) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:44,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 329 02.12.2021 17:25:44,574 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=685): org.apache.kafka.common.requests.HeartbeatResponse@38b83cad 02.12.2021 17:25:44,575 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:44,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 328 02.12.2021 17:25:44,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 328 02.12.2021 17:25:44,603 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:44,604 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:44,622 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119746ms for 120000ms of delay, sleeping for 264ms 02.12.2021 17:25:44,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 127 02.12.2021 17:25:44,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 228 02.12.2021 17:25:44,730 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59931ms for 60000ms of delay, sleeping for 79ms 02.12.2021 17:25:44,778 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 125 02.12.2021 17:25:44,768 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 23 02.12.2021 17:25:44,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,799 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,799 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,799 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,799 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,799 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,799 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,799 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,799 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:44,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,806 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:44,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,807 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,821 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59779ms for 60000ms of delay, sleeping for 231ms 02.12.2021 17:25:44,878 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 25 02.12.2021 17:25:44,888 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119515ms for 120000ms of delay, sleeping for 495ms 02.12.2021 17:25:44,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:44,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,904 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:44,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:44,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:44,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,945 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,991 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239803ms for 240000ms of delay, sleeping for 207ms 02.12.2021 17:25:45,001 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,002 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,002 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:45,008 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:45,057 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58825ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:45,064 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:45,064 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:45,064 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1409) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null} 02.12.2021 17:25:45,106 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:45,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:45,164 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1409): org.apache.kafka.common.requests.HeartbeatResponse@47dcf19e 02.12.2021 17:25:45,165 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:45,203 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239307ms for 240000ms of delay, sleeping for 703ms 02.12.2021 17:25:45,207 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:45,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:45,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,301 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,303 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,303 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,303 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,307 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,384 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118186ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:45,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,405 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,406 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:45,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:45,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,451 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,503 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,506 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:45,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:45,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:45,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:45,710 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:45,710 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:45,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:45,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,809 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,810 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:45,808 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,810 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,815 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,818 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 66 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 67 iterations. 02.12.2021 17:25:45,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:45,910 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 67 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:45,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,915 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,915 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 02.12.2021 17:25:45,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,916 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,916 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:45,917 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:45,917 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -108, 19, 23, -37, 87, 94, 65, -42, -84, -23, -18, 9, 35, 111, -82, -112, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:45,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:45,918 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1410) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:45,920 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=750): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:45,919 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963546, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:45,920 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963556, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:45,920 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', members=[]) 02.12.2021 17:25:45,920 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:45,920 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=692): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:45,921 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', members=[]) 02.12.2021 17:25:45,921 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:45,921 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:45,920 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:45,921 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:45,921 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=751) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null,assignments=[]} 02.12.2021 17:25:45,920 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1410): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:45,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 8 ms and fetched 10 records 02.12.2021 17:25:45,922 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:45,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,922 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:45,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 82 02.12.2021 17:25:45,921 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:45,923 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=693) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null,assignments=[]} 02.12.2021 17:25:45,923 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 386 02.12.2021 17:25:45,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 67 iterations. 02.12.2021 17:25:45,946 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239011ms for 240000ms of delay, sleeping for 999ms 02.12.2021 17:25:45,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,955 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,967 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:46,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:46,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 530 ms and fetched 0 records 02.12.2021 17:25:46,018 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:46,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,041 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,057 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59825ms delay out of 60000ms. Remaining 175ms. Executor queue at 0 02.12.2021 17:25:46,076 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59942ms for 60000ms of delay, sleeping for 68ms 02.12.2021 17:25:46,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:46,143 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 175 02.12.2021 17:25:46,170 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59860ms for 60000ms of delay, sleeping for 150ms 02.12.2021 17:25:46,183 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:46,183 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 191 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:46,183 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1155) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null} 02.12.2021 17:25:46,225 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:46,244 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 75 02.12.2021 17:25:46,284 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1155): org.apache.kafka.common.requests.HeartbeatResponse@4c97688e 02.12.2021 17:25:46,284 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during COMPLETING_REBALANCE state 02.12.2021 17:25:46,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:46,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,314 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:46,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,315 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:46,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:46,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:46,332 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155285, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:46,348 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155295, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:46,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 28 ms and fetched 10 records 02.12.2021 17:25:46,358 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:46,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,358 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:46,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:46,372 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59627ms for 60000ms of delay, sleeping for 383ms 02.12.2021 17:25:46,385 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119187ms delay out of 120000ms. Remaining 813ms. Executor queue at 0 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 72 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 149720ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 73 iterations. 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 73 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 149720ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:46,391 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Rebalance failed. 02.12.2021 17:25:46,391 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:46,391 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:46,391 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:46,391 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1156) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:46,393 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011952, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:46,396 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011962, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:46,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 6 ms and fetched 10 records 02.12.2021 17:25:46,399 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:46,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:46,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 73 iterations. 02.12.2021 17:25:46,399 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118926ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:46,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,417 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,423 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:46,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,459 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,459 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,488 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:46,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:46,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:46,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,544 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,544 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,544 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,545 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,618 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:46,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:46,721 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:46,724 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:46,781 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59956ms for 60000ms of delay, sleeping for 54ms 02.12.2021 17:25:46,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,815 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,815 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:46,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,824 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:46,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,827 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30350ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,828 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:46,840 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59781ms for 60000ms of delay, sleeping for 229ms 02.12.2021 17:25:46,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:46,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,928 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:46,929 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30466ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,953 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239701ms for 240000ms of delay, sleeping for 309ms 02.12.2021 17:25:46,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:46,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,961 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:47,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:47,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:47,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,047 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,047 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,077 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59374ms for 60000ms of delay, sleeping for 636ms 02.12.2021 17:25:47,092 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:47,092 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:47,092 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1071) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:47,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:47,132 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:47,193 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1071): org.apache.kafka.common.requests.HeartbeatResponse@67cb690 02.12.2021 17:25:47,193 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:47,232 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:47,232 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:47,265 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239654ms for 240000ms of delay, sleeping for 356ms 02.12.2021 17:25:47,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,316 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:47,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,327 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,328 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,330 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:47,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,331 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,331 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 231 02.12.2021 17:25:47,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:47,399 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119926ms delay out of 120000ms. Remaining 74ms. Executor queue at 0 02.12.2021 17:25:47,402 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119858ms for 120000ms of delay, sleeping for 152ms 02.12.2021 17:25:47,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:47,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,431 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,431 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 142 02.12.2021 17:25:47,432 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 131 02.12.2021 17:25:47,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,469 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=751): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,469 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=693): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,469 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,469 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,470 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7', protocol='stream'} 02.12.2021 17:25:47,469 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1', protocol='stream'} 02.12.2021 17:25:47,470 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 02.12.2021 17:25:47,470 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 02.12.2021 17:25:47,470 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:47,470 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:47,470 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:47,470 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:47,470 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:47,470 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:47,470 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:47,470 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:47,470 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from RUNNING to REBALANCING 02.12.2021 17:25:47,471 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:47,470 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:47,471 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:47,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 78 02.12.2021 17:25:47,471 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:47,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 348 02.12.2021 17:25:47,532 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 41 02.12.2021 17:25:47,532 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 30 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:47,549 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:47,549 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:47,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:47,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,550 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,556 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119734ms for 120000ms of delay, sleeping for 276ms 02.12.2021 17:25:47,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,563 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,564 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:47,564 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:47,564 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=757) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:47,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 267 02.12.2021 17:25:47,565 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=757): org.apache.kafka.common.requests.HeartbeatResponse@4e5d8b1d 02.12.2021 17:25:47,565 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:47,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 266 02.12.2021 17:25:47,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 266 02.12.2021 17:25:47,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 246 02.12.2021 17:25:47,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,575 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:47,577 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:47,577 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=686) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:47,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 356 02.12.2021 17:25:47,578 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=686): org.apache.kafka.common.requests.HeartbeatResponse@5828cbb3 02.12.2021 17:25:47,578 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:47,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 353 02.12.2021 17:25:47,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 353 02.12.2021 17:25:47,623 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238665ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:47,651 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:47,666 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 165 02.12.2021 17:25:47,672 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 145 02.12.2021 17:25:47,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 253 02.12.2021 17:25:47,723 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59946ms for 60000ms of delay, sleeping for 64ms 02.12.2021 17:25:47,752 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:47,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:47,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 150 02.12.2021 17:25:47,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 44 02.12.2021 17:25:47,795 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59726ms for 60000ms of delay, sleeping for 284ms 02.12.2021 17:25:47,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:47,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,832 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:47,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:47,834 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119789ms for 120000ms of delay, sleeping for 221ms 02.12.2021 17:25:47,832 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:47,836 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:47,836 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from REBALANCING to RUNNING 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,831 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,837 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,837 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,837 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,853 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:47,852 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,881 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 50 02.12.2021 17:25:47,932 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:47,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:47,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:47,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,940 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,940 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:47,954 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:47,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,965 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,971 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:48,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:48,038 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:48,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:48,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,054 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,055 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:48,060 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119280ms for 120000ms of delay, sleeping for 730ms 02.12.2021 17:25:48,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:48,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,095 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155295, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:48,095 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155305, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:48,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:48,100 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:48,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,100 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,100 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:48,104 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 57886ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:48,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:48,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 189 02.12.2021 17:25:48,145 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:48,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:48,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:48,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:48,251 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 85 02.12.2021 17:25:48,257 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:48,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,335 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,336 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:48,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,350 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:48,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:48,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 518 ms and fetched 0 records 02.12.2021 17:25:48,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,358 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,360 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:48,443 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:48,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:48,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,443 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,444 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:48,461 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:48,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,472 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,473 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:48,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:48,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:48,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,556 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 364 02.12.2021 17:25:48,623 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239665ms delay out of 240000ms. Remaining 335ms. Executor queue at 0 02.12.2021 17:25:48,624 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239360ms for 240000ms of delay, sleeping for 650ms 02.12.2021 17:25:48,644 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:48,646 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:48,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:48,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 264 02.12.2021 17:25:48,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:48,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:48,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:48,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 162 02.12.2021 17:25:48,795 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119682ms for 120000ms of delay, sleeping for 328ms 02.12.2021 17:25:48,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:48,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:48,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,850 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,850 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:48,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,853 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,853 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 68 02.12.2021 17:25:48,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 60 02.12.2021 17:25:48,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,860 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,921 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:48,921 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:48,921 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=752) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:48,922 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:48,922 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:48,922 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=694) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:48,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 135 02.12.2021 17:25:48,922 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=752): org.apache.kafka.common.requests.HeartbeatResponse@35aa89c6 02.12.2021 17:25:48,922 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:48,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 134 02.12.2021 17:25:48,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 134 02.12.2021 17:25:48,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 431 02.12.2021 17:25:48,923 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=694): org.apache.kafka.common.requests.HeartbeatResponse@4ff1623d 02.12.2021 17:25:48,923 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:48,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 430 02.12.2021 17:25:48,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 429 02.12.2021 17:25:48,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,951 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:48,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:48,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,974 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,974 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,022 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 34 02.12.2021 17:25:49,025 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 328 02.12.2021 17:25:49,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:49,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:49,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,057 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,105 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 58887ms delay out of 60000ms. Remaining 1113ms. Executor queue at 0 02.12.2021 17:25:49,106 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58738ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:49,124 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119997ms for 120000ms of delay, sleeping for 13ms 02.12.2021 17:25:49,126 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 227 02.12.2021 17:25:49,140 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119946ms for 120000ms of delay, sleeping for 64ms 02.12.2021 17:25:49,152 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:49,152 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:49,158 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:49,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:49,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:49,226 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 127 02.12.2021 17:25:49,258 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 73 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 152595ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 74 iterations. 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 74 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 152595ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:49,265 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011962, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:49,265 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011972, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:49,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:49,265 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:49,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,266 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:49,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 74 iterations. 02.12.2021 17:25:49,266 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119643ms for 120000ms of delay, sleeping for 367ms 02.12.2021 17:25:49,282 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239142ms for 240000ms of delay, sleeping for 868ms 02.12.2021 17:25:49,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:49,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,354 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,354 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,356 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:49,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:49,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,357 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,357 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:49,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,361 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:49,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,455 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:49,457 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:49,461 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:49,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,475 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:49,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:49,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:49,558 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,559 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,559 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,635 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119368ms for 120000ms of delay, sleeping for 642ms 02.12.2021 17:25:49,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:49,660 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:49,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:49,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:49,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:49,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:49,761 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:49,764 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:49,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:49,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,857 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,857 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,862 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:49,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,864 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:49,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,869 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:49,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:49,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,952 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,965 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:49,967 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 396 02.12.2021 17:25:49,967 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:49,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,977 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,982 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,053 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:50,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:50,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,065 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:50,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 290 02.12.2021 17:25:50,105 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,105 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:50,105 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1072) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:50,107 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59739ms delay out of 60000ms. Remaining 261ms. Executor queue at 1 02.12.2021 17:25:50,108 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59073ms for 60000ms of delay, sleeping for 937ms 02.12.2021 17:25:50,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:50,154 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239987ms for 240000ms of delay, sleeping for 23ms 02.12.2021 17:25:50,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:50,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:50,173 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 190 02.12.2021 17:25:50,180 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239947ms for 240000ms of delay, sleeping for 63ms 02.12.2021 17:25:50,205 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1072): org.apache.kafka.common.requests.HeartbeatResponse@d443970 02.12.2021 17:25:50,207 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:50,260 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239154ms for 240000ms of delay, sleeping for 856ms 02.12.2021 17:25:50,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:50,267 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:50,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:50,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 89 02.12.2021 17:25:50,284 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=765): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])]) 02.12.2021 17:25:50,285 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 24, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 91, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])]) 02.12.2021 17:25:50,299 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:50,299 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1442f54f, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1f87d9a1, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2e8dda8a, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@55d45efe, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@56bf2186, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7754b021, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@37cd7dcb, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4099f662, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6f855ce6, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3d022c8f, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@617d3dc1, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6ed8ed6a, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2b7fee3b, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7b5c98f, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@f2e6f8d, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@263460bf, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@726e4c74, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5b3e1a07, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@271044ea, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2224e92f, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@22a90cf3, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@9ecd483, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2c7ef640, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@29161f71, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@43325034, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2c15dc1d, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7782e329} 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,300 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:50,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Constructed client metadata {ce52c39d-7e35-47a3-9952-d434faf478ea=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 261b987a-3223-4915-a984-d667c12885af=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7bc41ba9-9d2e-4546-8593-56556fd30ef4=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 33919357-b8b1-49bb-aeb9-83ed9581975c=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 2d8e79dc-f788-4286-9c64-ec6210a1ff21=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 4fd6cb22-f124-4947-a2a1-f45a767078e7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 66529824-42fc-40c4-8d79-aa4feea5a732=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, cce182bf-5033-4aa1-bc49-50235c7a3ef3=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, f055f10f-2377-42c8-a04b-a6080c326d2b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:50,302 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:50,302 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:50,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:50,284 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=704): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:50,303 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:50,305 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119666ms for 120000ms of delay, sleeping for 344ms 02.12.2021 17:25:50,304 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:50,303 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:50,307 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:50,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:50,308 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Queueing Call(callName=metadata, deadlineMs=1638462410308, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:50,320 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:50,320 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462410308, tries=0, nextAllowedTryMs=0)] at 1638462350320 02.12.2021 17:25:50,320 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:50,321 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:50,307 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:50,321 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:50,320 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Metadata is ready to use. 02.12.2021 17:25:50,321 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Found least loaded node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) connected with no in-flight requests 02.12.2021 17:25:50,321 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Assigned Call(callName=metadata, deadlineMs=1638462410308, tries=0, nextAllowedTryMs=0) to node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) 02.12.2021 17:25:50,322 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-5.data.sta.redacted:9093 (id: 4 rack: null). correlationId=100 02.12.2021 17:25:50,322 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=100) and timeout 30000 to node 4: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:50,322 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59988) 02.12.2021 17:25:50,322 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:50,321 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:50,344 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=705) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,assignments=[]} 02.12.2021 17:25:50,321 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_1 from staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:50,345 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa due to tasks changing ownership. 02.12.2021 17:25:50,344 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462350344 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59964) 02.12.2021 17:25:50,345 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Received METADATA response from node 4 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=100): org.apache.kafka.common.requests.MetadataResponse@70776b0d 02.12.2021 17:25:50,354 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:50,345 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client ce52c39d-7e35-47a3-9952-d434faf478ea to schedule a followup rebalance 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 138 02.12.2021 17:25:50,354 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Call(callName=metadata, deadlineMs=1638462410308, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:50,356 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462350354 02.12.2021 17:25:50,356 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=137855) 02.12.2021 17:25:50,354 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client ce52c39d-7e35-47a3-9952-d434faf478ea per-consumer assignment: 02.12.2021 17:25:50,356 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 261b987a-3223-4915-a984-d667c12885af per-consumer assignment: 02.12.2021 17:25:50,356 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 7bc41ba9-9d2e-4546-8593-56556fd30ef4 per-consumer assignment: 02.12.2021 17:25:50,356 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 33919357-b8b1-49bb-aeb9-83ed9581975c per-consumer assignment: 02.12.2021 17:25:50,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:50,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,364 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:50,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:50,360 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_4 from staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:50,371 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 due to tasks changing ownership. 02.12.2021 17:25:50,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 to schedule a followup rebalance 02.12.2021 17:25:50,371 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 per-consumer assignment: 02.12.2021 17:25:50,371 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 4fd6cb22-f124-4947-a2a1-f45a767078e7 per-consumer assignment: 02.12.2021 17:25:50,371 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 66529824-42fc-40c4-8d79-aa4feea5a732 per-consumer assignment: 02.12.2021 17:25:50,372 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client cce182bf-5033-4aa1-bc49-50235c7a3ef3 per-consumer assignment: 02.12.2021 17:25:50,372 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client f055f10f-2377-42c8-a04b-a6080c326d2b per-consumer assignment: 02.12.2021 17:25:50,372 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:50,372 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Finished assignment for group at generation 192: {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry2-7], userDataSize=48), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-5], userDataSize=48), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-2], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-3], userDataSize=48), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-0], userDataSize=48), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-6], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-8], userDataSize=48)} 02.12.2021 17:25:50,372 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending leader SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:50,373 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=766) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]}]} 02.12.2021 17:25:50,367 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:50,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:50,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:50,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,380 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,378 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,382 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,386 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,389 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=766): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,388 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=705): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,390 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,390 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:50,389 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,394 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:50,394 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 192 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:50,391 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 192 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:50,395 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:50,395 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:50,394 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:50,397 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:50,397 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:50,397 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:50,397 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,397 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from RUNNING to REBALANCING 02.12.2021 17:25:50,397 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:50,395 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:50,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 491 02.12.2021 17:25:50,398 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:50,399 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,399 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:50,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:50,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 121 02.12.2021 17:25:50,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:50,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:50,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:50,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:50,491 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,491 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,492 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 390 02.12.2021 17:25:50,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 21 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,567 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:50,568 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=758) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:50,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:50,568 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=758): org.apache.kafka.common.requests.HeartbeatResponse@2e58f157 02.12.2021 17:25:50,568 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:50,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:50,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:50,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,574 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,575 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:50,577 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=687) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:50,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 379 02.12.2021 17:25:50,578 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=687): org.apache.kafka.common.requests.HeartbeatResponse@148203a3 02.12.2021 17:25:50,578 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:50,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 375 02.12.2021 17:25:50,579 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 374 02.12.2021 17:25:50,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:50,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:50,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 290 02.12.2021 17:25:50,651 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119482ms for 120000ms of delay, sleeping for 528ms 02.12.2021 17:25:50,697 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:50,689 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 191 02.12.2021 17:25:50,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 273 02.12.2021 17:25:50,668 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:50,668 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:50,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 178 02.12.2021 17:25:50,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:50,799 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 154 02.12.2021 17:25:50,799 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 269 02.12.2021 17:25:50,799 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 81 02.12.2021 17:25:50,799 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 66 02.12.2021 17:25:50,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 68 02.12.2021 17:25:50,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,866 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,867 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,867 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:50,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:50,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,882 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,893 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,893 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,894 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from REBALANCING to RUNNING 02.12.2021 17:25:50,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:50,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,894 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,894 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,900 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 168 02.12.2021 17:25:50,900 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:50,900 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 54 02.12.2021 17:25:50,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:50,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,957 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:50,968 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:50,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:50,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:50,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:50,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,994 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 68 02.12.2021 17:25:51,047 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58022ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:51,058 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:51,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:51,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,068 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:51,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:51,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:51,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:51,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 67 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:51,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 68 iterations. 02.12.2021 17:25:51,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 68 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:51,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:51,118 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', members=[]) 02.12.2021 17:25:51,118 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:51,118 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:51,118 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:51,118 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1411) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null,assignments=[]} 02.12.2021 17:25:51,119 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963556, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:51,119 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963566, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:51,119 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1411): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:51,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:51,120 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:51,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,120 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:51,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 68 iterations. 02.12.2021 17:25:51,121 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239872ms for 240000ms of delay, sleeping for 138ms 02.12.2021 17:25:51,158 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:51,168 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:51,169 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:51,181 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119996ms for 120000ms of delay, sleeping for 14ms 02.12.2021 17:25:51,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:51,195 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:51,196 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:51,196 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119953ms for 120000ms of delay, sleeping for 57ms 02.12.2021 17:25:51,219 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:51,226 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42', protocol='stream'} 02.12.2021 17:25:51,254 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119975ms for 120000ms of delay, sleeping for 35ms 02.12.2021 17:25:51,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:51,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:51,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:51,269 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239852ms for 240000ms of delay, sleeping for 158ms 02.12.2021 17:25:51,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:51,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:51,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:51,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 74 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:51,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 154638ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:51,309 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:51,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 75 iterations. 02.12.2021 17:25:51,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 75 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:51,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 154644ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:51,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:51,317 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:51,317 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1156): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:51,318 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', members=[]) 02.12.2021 17:25:51,319 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:51,319 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:51,320 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:51,320 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011972, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:51,320 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1157) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,assignments=[]} 02.12.2021 17:25:51,322 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011982, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:51,322 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1157): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:51,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 6 ms and fetched 10 records 02.12.2021 17:25:51,324 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:51,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,324 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:51,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 75 iterations. 02.12.2021 17:25:51,329 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119093ms for 120000ms of delay, sleeping for 917ms 02.12.2021 17:25:51,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:51,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,368 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,368 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,368 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,369 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:51,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,382 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,382 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:51,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:51,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:51,422 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=192, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', protocol='stream'} 02.12.2021 17:25:51,432 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239925ms for 240000ms of delay, sleeping for 85ms 02.12.2021 17:25:51,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,459 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,459 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:51,470 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:51,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:51,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 529 ms and fetched 0 records 02.12.2021 17:25:51,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,526 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,528 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239475ms for 240000ms of delay, sleeping for 535ms 02.12.2021 17:25:51,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:51,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,531 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,559 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:51,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,568 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 353 02.12.2021 17:25:51,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:51,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:51,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 271 02.12.2021 17:25:51,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:51,659 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:51,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:51,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 252 02.12.2021 17:25:51,683 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:51,734 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:51,732 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 167 02.12.2021 17:25:51,760 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:51,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:51,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 140 02.12.2021 17:25:51,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:51,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 58 02.12.2021 17:25:51,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 190 02.12.2021 17:25:51,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:51,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,869 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,869 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:51,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,884 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 33 02.12.2021 17:25:51,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 38 02.12.2021 17:25:51,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:51,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,923 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,929 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:51,931 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:51,931 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=695) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:51,929 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,932 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=695): org.apache.kafka.common.requests.HeartbeatResponse@10507bdd 02.12.2021 17:25:51,933 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:51,933 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 452 02.12.2021 17:25:51,933 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 452 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,959 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,960 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:51,961 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:51,961 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=753) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:51,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,961 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=753): org.apache.kafka.common.requests.HeartbeatResponse@7b5335da 02.12.2021 17:25:51,962 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:51,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 106 02.12.2021 17:25:51,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 89 02.12.2021 17:25:51,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 106 02.12.2021 17:25:51,969 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:52,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:52,034 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 352 02.12.2021 17:25:52,048 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59023ms delay out of 60000ms. Remaining 977ms. Executor queue at 0 02.12.2021 17:25:52,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 525 ms and fetched 0 records 02.12.2021 17:25:52,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,054 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59337ms for 60000ms of delay, sleeping for 673ms 02.12.2021 17:25:52,054 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,055 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:52,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 5 02.12.2021 17:25:52,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,069 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,069 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:52,085 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239101ms for 240000ms of delay, sleeping for 909ms 02.12.2021 17:25:52,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:52,135 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 250 02.12.2021 17:25:52,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:52,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:52,169 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:52,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:52,236 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 150 02.12.2021 17:25:52,249 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119943ms for 120000ms of delay, sleeping for 67ms 02.12.2021 17:25:52,257 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:52,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 294 02.12.2021 17:25:52,263 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:52,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:52,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:52,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 46 02.12.2021 17:25:52,367 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119957ms for 120000ms of delay, sleeping for 53ms 02.12.2021 17:25:52,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 183 02.12.2021 17:25:52,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:52,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,381 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,381 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,382 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,382 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,381 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:52,384 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 76 02.12.2021 17:25:52,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:52,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,396 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,425 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119605ms for 120000ms of delay, sleeping for 405ms 02.12.2021 17:25:52,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:52,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,428 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,462 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,463 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 74 02.12.2021 17:25:52,483 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 86 02.12.2021 17:25:52,483 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:52,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:52,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,556 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,556 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:52,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,570 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,570 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,584 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:52,598 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:52,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:52,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:52,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:52,672 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:52,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:52,698 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:52,728 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58652ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:52,729 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:52,757 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:52,768 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:52,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:52,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:52,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:52,830 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:52,834 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119986ms for 120000ms of delay, sleeping for 24ms 02.12.2021 17:25:52,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:52,867 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 117883ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:52,870 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:52,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 187 02.12.2021 17:25:52,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,894 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,894 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:52,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,901 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,928 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,929 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 370 02.12.2021 17:25:52,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,964 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,965 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:52,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 76 02.12.2021 17:25:53,001 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:53,004 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239707ms for 240000ms of delay, sleeping for 303ms 02.12.2021 17:25:53,029 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 270 02.12.2021 17:25:53,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,057 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 250 02.12.2021 17:25:53,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,085 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,085 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:53,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:53,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:53,129 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,129 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:53,129 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1073) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:53,132 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 167 02.12.2021 17:25:53,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 150 02.12.2021 17:25:53,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:53,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 277 02.12.2021 17:25:53,206 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:53,206 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:53,229 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1073): org.apache.kafka.common.requests.HeartbeatResponse@3346f542 02.12.2021 17:25:53,231 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:53,238 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 61 02.12.2021 17:25:53,258 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 49 02.12.2021 17:25:53,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:53,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 176 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,301 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 192 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,301 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=767) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null} 02.12.2021 17:25:53,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:53,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 128 02.12.2021 17:25:53,302 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=767): org.apache.kafka.common.requests.HeartbeatResponse@6e76973f 02.12.2021 17:25:53,302 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:53,302 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 192 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:53,302 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:53,303 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:53,303 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,303 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=768) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:53,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 126 02.12.2021 17:25:53,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:53,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:53,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:53,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:53,311 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,311 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 192 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,311 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=706) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null} 02.12.2021 17:25:53,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 246 02.12.2021 17:25:53,312 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239380ms for 240000ms of delay, sleeping for 630ms 02.12.2021 17:25:53,312 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=706): org.apache.kafka.common.requests.HeartbeatResponse@3bd1465 02.12.2021 17:25:53,312 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:53,312 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 192 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:53,312 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:53,312 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:53,312 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,312 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=707) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:53,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 244 02.12.2021 17:25:53,387 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:53,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:53,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,394 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 172 02.12.2021 17:25:53,401 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:53,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,402 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,412 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:53,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:53,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,430 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,430 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 110 02.12.2021 17:25:53,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:53,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:53,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:53,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,557 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 9 02.12.2021 17:25:53,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,567 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,568 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:53,568 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=759) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:53,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 328 02.12.2021 17:25:53,568 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=759): org.apache.kafka.common.requests.HeartbeatResponse@55bee7a9 02.12.2021 17:25:53,568 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:53,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 327 02.12.2021 17:25:53,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 326 02.12.2021 17:25:53,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,575 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,575 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:53,575 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=688) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:53,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 389 02.12.2021 17:25:53,575 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=688): org.apache.kafka.common.requests.HeartbeatResponse@65c23e3f 02.12.2021 17:25:53,575 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:53,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 389 02.12.2021 17:25:53,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 389 02.12.2021 17:25:53,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,588 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:53,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 226 02.12.2021 17:25:53,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 288 02.12.2021 17:25:53,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:53,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:53,728 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59652ms delay out of 60000ms. Remaining 348ms. Executor queue at 0 02.12.2021 17:25:53,733 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59812ms for 60000ms of delay, sleeping for 198ms 02.12.2021 17:25:53,770 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 125 02.12.2021 17:25:53,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:53,793 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:53,803 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:53,867 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 118883ms delay out of 120000ms. Remaining 1117ms. Executor queue at 1 02.12.2021 17:25:53,869 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 118744ms for 120000ms of delay, sleeping for 1000ms 02.12.2021 17:25:53,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 24 02.12.2021 17:25:53,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 81 02.12.2021 17:25:53,893 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:53,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,920 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,929 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:53,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,944 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239784ms for 240000ms of delay, sleeping for 226ms 02.12.2021 17:25:54,005 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59007ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:53,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,009 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,011 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:53,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,028 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,029 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,029 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,029 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,029 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,029 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,030 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:54,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:54,059 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:54,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,077 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,077 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 536 ms and fetched 0 records 02.12.2021 17:25:54,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,127 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:54,129 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:54,138 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,145 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:54,145 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1412) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null} 02.12.2021 17:25:54,135 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:54,148 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 294 02.12.2021 17:25:54,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 68 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 69 iterations. 02.12.2021 17:25:54,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 69 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:54,176 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1412): org.apache.kafka.common.requests.HeartbeatResponse@138c490f 02.12.2021 17:25:54,176 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:54,177 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 02.12.2021 17:25:54,177 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:54,177 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-2], userDataSize=48) 02.12.2021 17:25:54,177 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:54,177 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:330] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Skip resuming since state is RUNNING 02.12.2021 17:25:54,177 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:54,177 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:54,177 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from RUNNING to REBALANCING 02.12.2021 17:25:54,177 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [redacted.Cloud.Message.Callback.Retry3-2] 02.12.2021 17:25:54,177 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,180 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:54,180 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,180 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,180 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 496 02.12.2021 17:25:54,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:54,247 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:54,249 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:54,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:54,280 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,320 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 396 02.12.2021 17:25:54,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:54,338 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,348 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 192 and member id staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:54,348 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1158) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null} 02.12.2021 17:25:54,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 282 02.12.2021 17:25:54,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:54,360 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 170 02.12.2021 17:25:54,420 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,425 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 256 02.12.2021 17:25:54,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,431 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,436 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,449 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1158): org.apache.kafka.common.requests.HeartbeatResponse@548a339 02.12.2021 17:25:54,449 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during COMPLETING_REBALANCE state 02.12.2021 17:25:54,450 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 178 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 534 ms and fetched 0 records 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,454 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 475 02.12.2021 17:25:54,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 67 02.12.2021 17:25:54,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,514 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,514 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,515 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:54,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:54,532 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,533 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,533 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,529 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,533 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,533 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 148 02.12.2021 17:25:54,537 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:54,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 374 02.12.2021 17:25:54,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:54,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,578 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 310 02.12.2021 17:25:54,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,633 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 43 02.12.2021 17:25:54,637 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:54,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 271 02.12.2021 17:25:54,676 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-2 because it is paused 02.12.2021 17:25:54,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [redacted.Cloud.Message.Callback.Retry3-2] 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:54,677 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:54,677 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90] State transition from REBALANCING to RUNNING 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 69 iterations. 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 69 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:54,678 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963566, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:54,678 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963576, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:54,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:54,678 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:54,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,678 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:54,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 69 iterations. 02.12.2021 17:25:54,679 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239791ms for 240000ms of delay, sleeping for 219ms 02.12.2021 17:25:54,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 210 02.12.2021 17:25:54,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:54,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:54,758 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 171 02.12.2021 17:25:54,834 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 109 02.12.2021 17:25:54,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:54,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:54,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 69 02.12.2021 17:25:54,869 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 119744ms delay out of 120000ms. Remaining 256ms. Executor queue at 1 02.12.2021 17:25:54,875 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119621ms for 120000ms of delay, sleeping for 389ms 02.12.2021 17:25:54,899 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239832ms for 240000ms of delay, sleeping for 178ms 02.12.2021 17:25:54,931 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,931 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:54,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,934 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 9 02.12.2021 17:25:54,931 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=696) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:54,935 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=696): org.apache.kafka.common.requests.HeartbeatResponse@3c1944a 02.12.2021 17:25:54,935 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:54,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:54,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 19 02.12.2021 17:25:54,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 16 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,944 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,945 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:54,945 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=754) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:54,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,945 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=754): org.apache.kafka.common.requests.HeartbeatResponse@509a00d3 02.12.2021 17:25:54,945 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:54,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:54,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 187 02.12.2021 17:25:54,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,954 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,954 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:55,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:55,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:55,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:55,007 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155305, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:55,007 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155315, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:55,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:55,008 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:55,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,008 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:55,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:55,009 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58983ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:55,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,014 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,016 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:55,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,035 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,035 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,042 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:55,048 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 86 02.12.2021 17:25:55,055 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:55,079 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239012ms for 240000ms of delay, sleeping for 998ms 02.12.2021 17:25:55,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,079 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:55,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:55,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,139 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,139 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:55,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:55,236 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:55,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:55,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:55,256 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:55,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 75 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:55,266 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 158596ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 76 iterations. 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 76 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 158597ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:55,267 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Rebalance failed. 02.12.2021 17:25:55,267 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:55,267 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:55,268 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, correlationId=1159) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:55,269 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011982, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:55,269 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1011992, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:55,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:55,270 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:55,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,270 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:55,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 76 iterations. 02.12.2021 17:25:55,270 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119181ms for 120000ms of delay, sleeping for 829ms 02.12.2021 17:25:55,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:55,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:55,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:55,351 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:55,360 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:55,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:55,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,443 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,443 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,444 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:55,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:55,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,454 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,454 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,517 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,535 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,535 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,535 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,535 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,544 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:55,545 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:55,555 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:55,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,582 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:55,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,639 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,644 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:55,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:55,736 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:55,745 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:55,740 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:55,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:55,845 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 294 02.12.2021 17:25:55,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:55,845 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:55,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:55,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:55,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:55,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:55,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,951 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:55,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,963 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,010 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59984ms delay out of 60000ms. Remaining 16ms. Executor queue at 0 02.12.2021 17:25:56,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,018 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,018 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,026 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59771ms for 60000ms of delay, sleeping for 239ms 02.12.2021 17:25:56,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:56,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,046 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,048 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:56,048 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:56,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 389 02.12.2021 17:25:56,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:56,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:56,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,084 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,089 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,101 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239952ms for 240000ms of delay, sleeping for 58ms 02.12.2021 17:25:56,103 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119721ms for 120000ms of delay, sleeping for 289ms 02.12.2021 17:25:56,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:56,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,148 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,148 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:56,148 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:56,152 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:56,152 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1074) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:56,163 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:56,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:56,165 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239266ms for 240000ms of delay, sleeping for 744ms 02.12.2021 17:25:56,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:56,252 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1074): org.apache.kafka.common.requests.HeartbeatResponse@7e849fea 02.12.2021 17:25:56,252 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:56,251 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:56,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:56,266 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:56,268 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59442ms for 60000ms of delay, sleeping for 568ms 02.12.2021 17:25:56,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:56,364 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 184 02.12.2021 17:25:56,364 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:56,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:56,393 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119492ms for 120000ms of delay, sleeping for 518ms 02.12.2021 17:25:56,450 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 200 02.12.2021 17:25:56,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 114 02.12.2021 17:25:56,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,465 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,519 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:56,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,549 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 26 02.12.2021 17:25:56,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:56,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 14 02.12.2021 17:25:56,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:56,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,568 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:56,568 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:56,569 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=760) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:56,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 384 02.12.2021 17:25:56,569 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=760): org.apache.kafka.common.requests.HeartbeatResponse@4be2f979 02.12.2021 17:25:56,570 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:56,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 383 02.12.2021 17:25:56,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 383 02.12.2021 17:25:56,575 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:56,577 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:56,577 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=689) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:56,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 474 02.12.2021 17:25:56,578 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=689): org.apache.kafka.common.requests.HeartbeatResponse@1ce62b7e 02.12.2021 17:25:56,578 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:56,579 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 471 02.12.2021 17:25:56,579 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 470 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,650 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,651 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:56,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 283 02.12.2021 17:25:56,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 370 02.12.2021 17:25:56,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:56,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 182 02.12.2021 17:25:56,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:56,780 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 269 02.12.2021 17:25:56,846 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59929ms for 60000ms of delay, sleeping for 81ms 02.12.2021 17:25:56,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:56,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:56,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:56,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 164 02.12.2021 17:25:56,913 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119844ms for 120000ms of delay, sleeping for 166ms 02.12.2021 17:25:56,912 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239798ms for 240000ms of delay, sleeping for 212ms 02.12.2021 17:25:56,949 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59864ms for 60000ms of delay, sleeping for 146ms 02.12.2021 17:25:56,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,963 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:56,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,387 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,386 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,391 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:57,391 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1413) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer-c17fd77f-4288-4561-a331-0bdb167bca42,group_instance_id=null} 02.12.2021 17:25:57,386 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 736 ms and fetched 0 records 02.12.2021 17:25:57,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,392 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,392 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:57,386 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 921 ms and fetched 0 records 02.12.2021 17:25:57,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,395 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 10 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processed 0 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:57,405 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1155315, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:57,405 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1155325, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-1 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:57,405 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-1, new queue size is 10 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:57,406 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59444ms for 60000ms of delay, sleeping for 566ms 02.12.2021 17:25:57,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,407 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,407 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,407 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,407 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,407 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,407 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,407 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,020 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,408 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,990 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 59 02.12.2021 17:25:57,407 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:57,421 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239287ms for 240000ms of delay, sleeping for 723ms 02.12.2021 17:25:57,424 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119408ms for 120000ms of delay, sleeping for 602ms 02.12.2021 17:25:57,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 919 ms and fetched 0 records 02.12.2021 17:25:57,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,485 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:57,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:57,491 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, correlationId=1413): org.apache.kafka.common.requests.HeartbeatResponse@22ce8dc2 02.12.2021 17:25:57,505 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:57,515 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 380 02.12.2021 17:25:57,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:57,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:57,598 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:57,622 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 273 02.12.2021 17:25:57,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:57,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:57,699 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:57,723 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 172 02.12.2021 17:25:57,791 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:57,794 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:57,800 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:57,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 69 02.12.2021 17:25:57,888 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 925 ms and fetched 0 records 02.12.2021 17:25:57,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,895 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,892 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,897 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,897 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 47 02.12.2021 17:25:57,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:57,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:57,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,907 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:57,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,911 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,907 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 24 02.12.2021 17:25:57,912 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,916 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,936 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,940 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:57,940 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=697) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer-9da0791a-ace1-4142-80a6-9179bbd32cc7,group_instance_id=null} 02.12.2021 17:25:57,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 471 02.12.2021 17:25:57,945 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,943 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, correlationId=697): org.apache.kafka.common.requests.HeartbeatResponse@228584c 02.12.2021 17:25:57,948 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:57,948 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 459 02.12.2021 17:25:57,948 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 459 02.12.2021 17:25:57,945 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,949 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:57,949 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=755) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer-f1163362-b85c-46db-a91c-12b88f1cbde1,group_instance_id=null} 02.12.2021 17:25:57,949 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 452 02.12.2021 17:25:57,956 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, correlationId=755): org.apache.kafka.common.requests.HeartbeatResponse@45ebf7c3 02.12.2021 17:25:57,958 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:57,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 439 02.12.2021 17:25:57,959 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 438 02.12.2021 17:25:57,976 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59980ms for 60000ms of delay, sleeping for 30ms 02.12.2021 17:25:57,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:57,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,992 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:57,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,027 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119991ms for 120000ms of delay, sleeping for 19ms 02.12.2021 17:25:58,048 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 359 02.12.2021 17:25:58,052 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119423ms for 120000ms of delay, sleeping for 587ms 02.12.2021 17:25:58,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 338 02.12.2021 17:25:58,068 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59940ms for 60000ms of delay, sleeping for 70ms 02.12.2021 17:25:58,098 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:58,101 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 391 02.12.2021 17:25:58,144 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59990ms for 60000ms of delay, sleeping for 20ms 02.12.2021 17:25:58,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 258 02.12.2021 17:25:58,158 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239979ms for 240000ms of delay, sleeping for 31ms 02.12.2021 17:25:58,160 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 238 02.12.2021 17:25:58,176 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59200ms for 60000ms of delay, sleeping for 810ms 02.12.2021 17:25:58,191 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239235ms for 240000ms of delay, sleeping for 775ms 02.12.2021 17:25:58,201 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:58,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:58,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 137 02.12.2021 17:25:58,256 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 152 02.12.2021 17:25:58,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:58,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:58,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 34 02.12.2021 17:25:58,372 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 36 02.12.2021 17:25:58,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:58,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,395 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,398 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:58,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,408 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,408 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,417 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:58,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,492 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,492 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:58,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:58,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:58,593 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:58,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:58,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:58,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:58,641 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119511ms for 120000ms of delay, sleeping for 499ms 02.12.2021 17:25:58,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:58,697 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:58,699 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:58,710 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:58,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:58,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:58,799 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:58,824 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 84 02.12.2021 17:25:58,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,897 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:58,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,904 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,917 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,917 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,918 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,924 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,926 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 10 records with 69 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:58,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 70 iterations. 02.12.2021 17:25:58,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processed 0 records with 70 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:58,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:58,977 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=963576, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:58,978 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=963586, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-2 02.12.2021 17:25:58,978 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:58,979 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-2, new queue size is 10 02.12.2021 17:25:58,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,979 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:58,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-1] Processing tasks with 70 iterations. 02.12.2021 17:25:58,981 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239789ms for 240000ms of delay, sleeping for 221ms 02.12.2021 17:25:58,989 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58996ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:58,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:58,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,996 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,996 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,998 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:59,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:59,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:59,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:59,099 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:59,106 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:59,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 10 records with 76 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 162471ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 77 iterations. 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processed 0 records with 77 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Committing all active tasks [0_2] and standby tasks [] since 162471ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:59,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:59,141 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Returning 10 fetched records at offset FetchPosition{offset=1011992, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:59,141 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Update fetching position to FetchPosition{offset=1012002, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-5.data.sta.redacted:9093 (id: 4 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry2-2 02.12.2021 17:25:59,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:59,142 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] task [0_2] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry2-2, new queue size is 10 02.12.2021 17:25:59,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,142 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:59,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1] Processing tasks with 77 iterations. 02.12.2021 17:25:59,143 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 119049ms for 120000ms of delay, sleeping for 961ms 02.12.2021 17:25:59,194 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:59,194 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:59,194 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1075) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer-b056d751-2db7-49c2-9766-d9dbe4a129aa,group_instance_id=null} 02.12.2021 17:25:59,197 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:59,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:59,203 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239998ms for 240000ms of delay, sleeping for 12ms 02.12.2021 17:25:59,206 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:59,218 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:59,218 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239631ms for 240000ms of delay, sleeping for 379ms 02.12.2021 17:25:59,294 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, correlationId=1075): org.apache.kafka.common.requests.HeartbeatResponse@1d2f6e9c 02.12.2021 17:25:59,296 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:59,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:59,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:59,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:59,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:59,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:59,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:59,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,404 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:59,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,405 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,405 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,405 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 163 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:59,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,427 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,427 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:59,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,432 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,497 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 78 02.12.2021 17:25:59,505 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:59,505 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 63 02.12.2021 17:25:59,510 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=768): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 25, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 25, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 25, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])]) 02.12.2021 17:25:59,510 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=707): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:59,515 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', members=[JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 25, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 1]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 25, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 123, -60, 27, -87, -99, 46, 69, 70, -123, -109, 86, 85, 111, -45, 14, -12, 0, 0, 0, 0, 0, 0, 0, 0, 25, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 38, 27, -104, 122, 50, 35, 73, 21, -87, -124, -42, 103, -63, 40, -123, -81, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 4]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 79, -42, -53, 34, -15, 36, 73, 71, -94, -95, -12, 90, 118, 112, 120, -25, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -16, 85, -15, 15, 35, 119, 66, -56, -96, 75, -90, 8, 12, 50, 109, 43, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -52, -31, -126, -65, 80, 51, 74, -95, -68, 73, 80, 35, 92, 122, 62, -13, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -50, 82, -61, -99, 126, 53, 71, -93, -103, 82, -44, 52, -6, -12, 120, -22, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 102, 82, -104, 36, 66, -4, 64, -60, -115, 121, -86, 79, -18, -91, -89, 50, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 51, -111, -109, 87, -72, -79, 73, -69, -82, -71, -125, -19, -107, -127, -105, 92, 0, 0, 0, 0, 0, 0, 0, 0, 92, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5]), JoinGroupResponseMember(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])]) 02.12.2021 17:25:59,515 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:59,515 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=193, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:59,515 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', members=[]) 02.12.2021 17:25:59,516 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:59,516 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=193, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:59,516 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=193, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=193, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:59,516 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=708) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=193,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,group_instance_id=null,assignments=[]} 02.12.2021 17:25:59,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 417 02.12.2021 17:25:59,516 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:587] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Performing assignment using strategy stream with subscriptions {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3f14d935, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3d630ffe, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5d9838d0, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@78d9333, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1e492f86, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3b70df4e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@27b22e48, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@4c4d3586, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1baf0696, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@26182002, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@106e8ae7, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@352a7c69, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@56d257ec, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@312d7504, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@55f0e874, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@335f1f97, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@77f64327, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2da98d88, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@7fc87301, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@34a068c8, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@710afa00, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@10aefc64, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@2c42d50b, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@1b930e63, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@3d6f3deb, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@5a45bbfe, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=org.apache.kafka.clients.consumer.ConsumerPartitionAssignor$Subscription@6015c730} 02.12.2021 17:25:59,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1567] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Current minimum supported version remains at 8, last seen supported version was 8 02.12.2021 17:25:59,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:343] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Constructed client metadata {ce52c39d-7e35-47a3-9952-d434faf478ea=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e, staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 261b987a-3223-4915-a984-d667c12885af=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1, staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 7bc41ba9-9d2e-4546-8593-56556fd30ef4=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf, staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 33919357-b8b1-49bb-aeb9-83ed9581975c=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd, staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 2d8e79dc-f788-4286-9c64-ec6210a1ff21=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23, staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 4fd6cb22-f124-4947-a2a1-f45a767078e7=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a, staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, 66529824-42fc-40c4-8d79-aa4feea5a732=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635, staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, cce182bf-5033-4aa1-bc49-50235c7a3ef3=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e, staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}, f055f10f-2377-42c8-a04b-a6080c326d2b=ClientMetadata{hostInfo=null, consumers=[staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c, staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638], state=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]}} from the member subscriptions. 02.12.2021 17:25:59,519 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:59,519 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:59,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:356] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created repartition topics [] from the parsed topology. 02.12.2021 17:25:59,520 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:105] stream-thread [main] Starting to validate internal topics {} in partition assignor. 02.12.2021 17:25:59,520 DEBUG [org.apache.kafka.streams.processor.internals.InternalTopicManager:193] stream-thread [main] Completed validating internal topics and created [] 02.12.2021 17:25:59,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:693] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Created state changelog topics [] from the parsed topology. 02.12.2021 17:25:59,520 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1358] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Queueing Call(callName=metadata, deadlineMs=1638462419520, tries=0, nextAllowedTryMs=0) with a timeout 60000 ms from now. 02.12.2021 17:25:59,520 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:59,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:734] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigning tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] to clients {ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 0]} with number of replicas 0 02.12.2021 17:25:59,520 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [Call(callName=metadata, deadlineMs=1638462419520, tries=0, nextAllowedTryMs=0)] at 1638462359520 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.admin.internals.AdminMetadataManager:159] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Metadata is ready to use. 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.NetworkClient:690] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Found least loaded node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) connected with no in-flight requests 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1043] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Assigned Call(callName=metadata, deadlineMs=1638462419520, tries=0, nextAllowedTryMs=0) to node kafka-5.data.sta.redacted:9093 (id: 4 rack: null) 02.12.2021 17:25:59,521 DEBUG [org.apache.kafka.clients.admin.KafkaAdminClient:1094] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-5.data.sta.redacted:9093 (id: 4 rack: null). correlationId=101 02.12.2021 17:25:59,521 DEBUG [org.apache.kafka.clients.NetworkClient:522] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=101) and timeout 30000 to node 4: {topics=[],allow_auto_topic_creation=false,include_cluster_authorized_operations=false,include_topic_authorized_operations=false} 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=60000) 02.12.2021 17:25:59,521 INFO [org.apache.kafka.streams.processor.internals.assignment.HighAvailabilityTaskAssignor:95] Decided on assignment: {cce182bf-5033-4aa1-bc49-50235c7a3ef3=[activeTasks: ([0_0]) standbyTasks: ([]) prevActiveTasks: ([0_1]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], ce52c39d-7e35-47a3-9952-d434faf478ea=[activeTasks: ([0_1]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], f055f10f-2377-42c8-a04b-a6080c326d2b=[activeTasks: ([0_2]) standbyTasks: ([]) prevActiveTasks: ([0_2]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 261b987a-3223-4915-a984-d667c12885af=[activeTasks: ([0_3]) standbyTasks: ([]) prevActiveTasks: ([0_4]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 2d8e79dc-f788-4286-9c64-ec6210a1ff21=[activeTasks: ([0_4]) standbyTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 33919357-b8b1-49bb-aeb9-83ed9581975c=[activeTasks: ([0_5]) standbyTasks: ([]) prevActiveTasks: ([0_5]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 4fd6cb22-f124-4947-a2a1-f45a767078e7=[activeTasks: ([0_6]) standbyTasks: ([]) prevActiveTasks: ([0_6]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 66529824-42fc-40c4-8d79-aa4feea5a732=[activeTasks: ([0_7]) standbyTasks: ([]) prevActiveTasks: ([0_7]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1], 7bc41ba9-9d2e-4546-8593-56556fd30ef4=[activeTasks: ([0_8]) standbyTasks: ([]) prevActiveTasks: ([0_8]) prevStandbyTasks: ([]) changelogOffsetTotalsByTask: ([]) taskLagTotals: ([]) capacity: 3 assigned: 1]} with no followup probing rebalance. 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 0 response(s) 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462359521 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=59999) 02.12.2021 17:25:59,521 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:744] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Assigned tasks [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8] including stateful [] to clients as: 02.12.2021 17:25:59,521 DEBUG [org.apache.kafka.clients.NetworkClient:884] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Received METADATA response from node 4 for request with header RequestHeader(apiKey=METADATA, apiVersion=8, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin, correlationId=101): org.apache.kafka.common.requests.MetadataResponse@63a38acf 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1330] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] KafkaClient#poll retrieved 1 response(s) 02.12.2021 17:25:59,521 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_1 from staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:59,521 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1186] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Call(callName=metadata, deadlineMs=1638462419520, tries=0, nextAllowedTryMs=0) got response {throttle_time_ms=0,brokers=[{node_id=2,host=kafka-3.data.sta.redacted,port=9093,rack=null},{node_id=3,host=kafka-4.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null},{node_id=4,host=kafka-5.data.sta.par.eu.dc.sigfox.com,port=9093,rack=null}],cluster_id=8sIXt2WcSU2XFEbFh5v__g,controller_id=2,topics=[],cluster_authorized_operations=0} 02.12.2021 17:25:59,521 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa due to tasks changing ownership. 02.12.2021 17:25:59,522 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1018] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Trying to choose nodes for [] at 1638462359521 02.12.2021 17:25:59,522 TRACE [org.apache.kafka.clients.admin.KafkaAdminClient:1328] [AdminClient clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-admin] Entering KafkaClient#poll(timeout=128688) 02.12.2021 17:25:59,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client ce52c39d-7e35-47a3-9952-d434faf478ea to schedule a followup rebalance 02.12.2021 17:25:59,522 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client ce52c39d-7e35-47a3-9952-d434faf478ea per-consumer assignment: 02.12.2021 17:25:59,522 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 261b987a-3223-4915-a984-d667c12885af per-consumer assignment: 02.12.2021 17:25:59,522 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 7bc41ba9-9d2e-4546-8593-56556fd30ef4 per-consumer assignment: 02.12.2021 17:25:59,522 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 33919357-b8b1-49bb-aeb9-83ed9581975c per-consumer assignment: 02.12.2021 17:25:59,522 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1115] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Removing task 0_4 from staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 active assignment until it is safely revoked in followup rebalance 02.12.2021 17:25:59,522 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1063] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requesting followup rebalance be scheduled immediately by staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 due to tasks changing ownership. 02.12.2021 17:25:59,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:975] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Requested client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 to schedule a followup rebalance 02.12.2021 17:25:59,523 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 2d8e79dc-f788-4286-9c64-ec6210a1ff21 per-consumer assignment: 02.12.2021 17:25:59,523 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 4fd6cb22-f124-4947-a2a1-f45a767078e7 per-consumer assignment: 02.12.2021 17:25:59,523 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client 66529824-42fc-40c4-8d79-aa4feea5a732 per-consumer assignment: 02.12.2021 17:25:59,523 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client cce182bf-5033-4aa1-bc49-50235c7a3ef3 per-consumer assignment: 02.12.2021 17:25:59,523 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:978] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Client f055f10f-2377-42c8-a04b-a6080c326d2b per-consumer assignment: 02.12.2021 17:25:59,523 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:994] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] Finished unstable assignment of tasks, a followup rebalance will be scheduled. 02.12.2021 17:25:59,523 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:626] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Finished assignment for group at generation 193: {staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262=Assignment(partitions=[redacted.Cloud.Message.Callback.Retry2-7], userDataSize=48), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-5], userDataSize=48), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-2], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-3], userDataSize=48), staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-0], userDataSize=48), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-6], userDataSize=48), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1=Assignment(partitions=[], userDataSize=40), staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336=Assignment(partitions=[OSS.STAGING.Cloud.Message.Callback.Retry2-8], userDataSize=48)} 02.12.2021 17:25:59,524 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:711] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending leader SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=193, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=193, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 7, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 3, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 6, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1', assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]), SyncGroupRequestAssignment(memberId='staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336', assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1])]) 02.12.2021 17:25:59,524 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=769) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=193,member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,group_instance_id=null,assignments=[{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-1-consumer-3ab1ad8c-7553-4615-8489-33529f5ab262,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-3-consumer-4c69ca67-635e-42ec-8cca-ed8d0138f4cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-2-consumer-1959a070-2423-45c3-a3c8-9fba796e2d3e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-1-consumer-9e46622f-af12-4e8c-a662-7163ece6e6fd,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-3-consumer-701a68e4-aed9-4943-afcc-9aa1b58e9e39,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-66529824-42fc-40c4-8d79-aa4feea5a732-StreamThread-2-consumer-19180362-624c-4921-a71f-db9dfe534635,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-3-consumer-0df95d64-47d7-4228-a15d-80e6cc336101,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-1-consumer-b83e36ed-a590-4895-84ef-1ad9d094314e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-1-consumer-fbe407e7-3afd-461f-9cf4-ba856c356cd6,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-33919357-b8b1-49bb-aeb9-83ed9581975c-StreamThread-2-consumer-02cd3c2a-259c-4bfd-b98c-21f5fdd257cd,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-1-consumer-b3db5b45-b22e-4ca1-8c2b-f83c787140e4,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-3-consumer-3b941dd0-1bc6-4e8e-b050-e0d0ddc76d82,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-cce182bf-5033-4aa1-bc49-50235c7a3ef3-StreamThread-3-consumer-53ac5ce0-181a-4597-b343-4224d37d3bd2,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-2-consumer-68eb7edd-d1ad-47e8-91f9-c99837ad0e6e,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-2-consumer-a7884997-f917-4122-9c6a-7908d248db9a,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-2-consumer-74936464-9c6d-4ebf-9c2a-7b561f295fcf,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-3-consumer-f6a3db3b-3dd4-4cef-b1f1-cd5dd29a8d28,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-4fd6cb22-f124-4947-a2a1-f45a767078e7-StreamThread-1-consumer-a3e966fa-c26c-4f09-bc87-7eed96ada69e,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-3-consumer-7dee4c04-66b3-4795-bd16-153354eb7328,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-ce52c39d-7e35-47a3-9952-d434faf478ea-StreamThread-1-consumer-6edb49f2-3c79-4f88-acdc-44e018549cfa,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-261b987a-3223-4915-a984-d667c12885af-StreamThread-2-consumer-211363ff-105c-4646-aed1-391ec5af1ca1,assignment=java.nio.HeapByteBuffer[pos=0 lim=50 cap=50]},{member_id=staging-message-processor-retry2-7bc41ba9-9d2e-4546-8593-56556fd30ef4-StreamThread-1-consumer-fe266161-af09-47a3-952a-f2d90c33f336,assignment=java.nio.HeapByteBuffer[pos=0 lim=109 cap=109]}]} 02.12.2021 17:25:59,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 402 02.12.2021 17:25:59,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 397 02.12.2021 17:25:59,542 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, correlationId=769): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,542 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, correlationId=708): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,542 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,542 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=193, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', protocol='stream'} 02.12.2021 17:25:59,543 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 193 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c 02.12.2021 17:25:59,543 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:59,543 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:59,543 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:59,543 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:59,543 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:59,543 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from RUNNING to REBALANCING 02.12.2021 17:25:59,543 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:59,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 385 02.12.2021 17:25:59,542 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,543 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=193, memberId='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638', protocol='stream'} 02.12.2021 17:25:59,544 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 193 and memberId staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer-ea50fff9-3173-46b7-8e5a-62fb173ff638 02.12.2021 17:25:59,545 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:59,545 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:59,545 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:59,545 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:59,546 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:59,546 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:59,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 391 02.12.2021 17:25:59,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:59,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:59,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:59,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:59,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:59,569 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:59,570 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:59,570 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=761) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer-8c17baa7-8dd6-4eb1-814e-62d2cf461d63,group_instance_id=null} 02.12.2021 17:25:59,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 336 02.12.2021 17:25:59,572 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, correlationId=761): org.apache.kafka.common.requests.HeartbeatResponse@6945867b 02.12.2021 17:25:59,572 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:59,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 333 02.12.2021 17:25:59,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 333 02.12.2021 17:25:59,575 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:59,578 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:59,578 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=690) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer-66af8107-6197-42fb-a6c7-63211ccc1957,group_instance_id=null} 02.12.2021 17:25:59,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 422 02.12.2021 17:25:59,585 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, correlationId=690): org.apache.kafka.common.requests.HeartbeatResponse@20d1e02 02.12.2021 17:25:59,589 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:59,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 408 02.12.2021 17:25:59,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 408 02.12.2021 17:25:59,603 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239370ms for 240000ms of delay, sleeping for 640ms 02.12.2021 17:25:59,605 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:59,628 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:59,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 284 02.12.2021 17:25:59,646 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 287 02.12.2021 17:25:59,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 232 02.12.2021 17:25:59,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 307 02.12.2021 17:25:59,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:59,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 194 02.12.2021 17:25:59,744 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 183 02.12.2021 17:25:59,749 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 184 02.12.2021 17:25:59,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 132 02.12.2021 17:25:59,790 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 207 02.12.2021 17:25:59,806 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:59,842 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 83 02.12.2021 17:25:59,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 84 02.12.2021 17:25:59,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 78 02.12.2021 17:25:59,876 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 29 02.12.2021 17:25:59,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 105 02.12.2021 17:25:59,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:59,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,904 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,906 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,913 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,914 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:59,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:59,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,925 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-941317db-575e-41d6-ace9-ee09236fae90-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,934 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:59,934 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:59,935 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:59,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:59,936 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:59,935 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,936 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:59,942 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:59,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,942 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:59,953 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:59,953 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:59,958 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:59,958 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b] State transition from REBALANCING to RUNNING 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-adf61f72-5987-4364-b122-719e04c03d07-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 5 02.12.2021 17:25:59,989 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59996ms delay out of 60000ms. Remaining 4ms. Executor queue at 0